iFLYTEK unveils iFLYTEK-Embodied-Omni, a unified multimodal foundation model for embodied agents

General-purpose embodied agents must interpret multimodal instructions, anticipate environmental changes, and execute precise control actions over extended horizons. Existing approaches often specialize in visual-language reasoning, video-based world modeling, or action generation, and cascaded pipelines that first predict future observations and then infer actions can introduce interface bottlenecks and compound prediction errors.

iFLYTEK-Embodied-Omni is a unified multimodal foundation model that jointly models vision (videos and images), language, and action within a single Omni framework. Its modality-specific components—visual-language, video-generation, and action-generation—communicate via shared multimodal self-attention.

The design establishes a brain-cerebellum collaboration: the vision-language and video-generation models act as a high-level brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, while the action-generation model serves as a low-level cerebellum that converts planned subgoals and shared multimodal context into executable action chunks.

To train the model, the authors combine action-annotated and action-free embodied videos from human demonstrations and robot interactions with embodied reasoning, embodied perception, and general-purpose image-text data to construct a comprehensive dataset. They adopt a four-stage strategy that progressively trains the visual-language model, video-generation model, and action-generation model before jointly fine-tuning the complete system.