A recent paper from the Orca Team at the Beijing Academy of Artificial Intelligence has floated a new way of approaching the traditional AI learning model. Their model, called Orca, is presented as an early world foundation model designed to learn from interactions across different forms of data instead of relying on a single modality. Instead of treating language, images and actions as separate prediction problems, the researchers focus on predicting how the state of the world changes over time.
This article walks through what Orca is, how its architecture differs from conventional large models, how it was trained, what the results show and just as importantly what those results don't yet prove.
Why Orca required rethinking "prediction"
Most large AI models today fall into three categories. Language models predict the next token, image models generate the next frame, and robotics models predict the next action. While each performs well in its own domain, they do not develop a shared understanding of how the world changes over time.
The Orca researchers propose a different approach called Next-State Prediction. Instead of learning separate patterns for text, images and actions, Orca builds a unified internal representation called the World Latent. This shared representation captures changes in the world and can later be used to generate language, images or actions depending on the task. The goal is to model the underlying dynamics of the world rather than individual data modalities.
The Orca architecture
Orca follows a straightforward Encoder–Decoder layout, yet the split of responsibilities is what makes it distinctive.
Encoder
Orca begins with an encoder built on a vision-language backbone that processes both visual and text inputs together. Instead of keeping these signals separate, the encoder combines them into a compact internal representation called the World Latent. This latent space is designed to capture what is happening in a scene without being tied to how that understanding will later be expressed, whether through text, images, or actions.
The model learns this representation through two complementary training processes.
Unconscious learning: It uses large volumes of unlabeled video. By predicting the latent representation of the next moment rather than the next image itself, Orca learns how scenes change over time. This helps it model physical behavior such as motion, object interactions, occlusion, and changes in the environment without relying on manual annotations.
Conscious learning: It uses a smaller set of labeled video examples. Here, text descriptions such as "close the microwave door" provide explicit context about what is happening in a scene. These labels encourage the model to associate changes in the world with meaningful events instead of only tracking visual differences between frames. At the same time, a visual question-answering objective helps maintain its ability to understand language and reason about everyday situations.
Decoder
After pretraining is complete, the researchers freeze the backbone and leave its parameters unchanged. From that point onward, they train only lightweight decoders for individual tasks such as language, vision, or action.
Three readout paths are examined:
- Text: Reusing the existing language-model head to translate the latent directly into natural sentences.
- Image: Routing the latent through a small adapter and a LoRA-tuned Stable Diffusion 3.5 module to visualize predicted future frames.
- Action: Feeding the latent into a diffusion-style Action Expert alongside robot proprioception, producing motor commands for a dual-arm humanoid platform.
Enterprises exploring similar multimodal architectures often need help translating research-stage models into deployable systems a gap GoML's AI blueprints are designed to close
The data behind Orca
Training Orca required a large and varied dataset. The researchers gathered about 125,000 hours of video covering first-person interactions, third-person object manipulation, unlabeled robot demonstrations, and everyday scenes. They paired this with around 160 million event captions, ranging from broad activity descriptions to finer action transitions, along with 11.5 million visual question-answer pairs that helped strengthen the model's connection between visual inputs and language.
One detail stands out in the paper. The current 4B-parameter model was trained on only about one-tenth of the collected dataset, while the remaining data is reserved for future training runs. That means the reported results reflect only a portion of the available data. As training expands, both the model's strengths and its current limitations may change.
Putting Orca mandatory claims to the test
The researchers organized their evaluation around two main questions. First, does the training approach continue to improve as the model size and dataset increase? Second, does a better World Latent lead to stronger performance across downstream tasks?
Scaling behavior
Training results for both the 0.8 billion and 4 billion parameter models showed a consistent decline in loss as additional video data was introduced. The researchers did not observe signs of early saturation, suggesting that the approach continues to improve with larger datasets instead of reaching a performance ceiling too quickly.
Text generation
Across four benchmarks covering temporal reasoning, spatial understanding, and general video comprehension, the 4 billion parameter version of Orca achieved stronger results than similarly sized vision-language models such as Qwen3.5 and Gemma 4. It also outperformed larger world models, including Emu3.5, despite having fewer parameters. The evaluation also showed particularly strong performance in predicting state transitions and handling commonsense reasoning tasks.
Image prediction
To evaluate visual prediction under realistic conditions, the researchers introduced a new benchmark called PRICE-V0.1, built from real-world interaction videos rather than synthetic scenes. Independent multimodal evaluators compared the generated predictions and found that Orca produced more accurate future frames than similarly sized image generation models such as FLUX.2 and OmniGen2. The model also produced fewer common errors, including objects appearing in impossible locations and unrealistic hand positions.
Embodied action
The researchers evaluated Orca on five dual-arm robotic tasks, including unfamiliar environments and objects. It outperformed a comparable vision-language model and matched the performance of a robotics policy trained on action-labeled data, despite never using action annotations during pretraining. Orca also recovered from failed grasping attempts more effectively, allowing it to complete tasks more reliably.
Ablation insights
The ablation studies showed that each training objective contributed a distinct capability. Removing the unconditioned transition objective reduced action performance, removing the event-conditioned objective weakened image prediction, and removing the visual question-answering objective affected language understanding. The best overall performance came from combining all three objectives.
Where Orca doesn't do yet
The researchers are also clear about the current limitations of Orca:
- The model learns only from vision and language. It does not use audio, touch, or force signals, even though those inputs can reveal changes in the environment before they become visible.
- The World Latent representation is tied to the embedding space of an existing vision encoder instead of being learned independently. This could limit how much it moves beyond the assumptions of that encoder.
- The current 4B-parameter model uses only a portion of the available training data, leaving plenty of room to test how the approach performs at a larger scale.
- Event annotations focus on short-term activities, so there is little evidence yet of how well the model handles long-term planning.
- Robotic evaluations cover relatively simple tasks rather than more demanding real-world scenarios.
Overall, Orca introduces a training method that continues improving as more data is added without showing the early plateau seen in many existing models. It also performs well on both the benchmarks introduced in the paper and the evaluation tasks selected by the authors. Even so, the approach still needs broader testing across larger models, more demanding tasks, and independent benchmarks before its performance can be compared confidently with today's leading AI systems.
What comes next
The researchers plan to add sensory inputs like audio and touch, train a version with its own native representation rather than one anchored to an existing vision encoder, and build benchmarks specifically for state-transition reasoning. They also describe a self-improving training loop in which the model would learn from its own simulated experience. Longer-term, they see this approach extending beyond robotics into scientific and life-sciences domains where modeling how systems change over time matters.
Closing thought on Orca
Orca is not presented as a complete solution for general intelligence, and it does not lead every benchmark. Its main contribution is showing that a single shared internal representation, learned by predicting how the world changes over time, can support language, image generation, and physical actions within the same model.
If this approach continues to improve with larger models and datasets, Next-State Prediction could become an important direction for building future foundation models. That's where a structured framework matters. GoML's AI Matic is built to help enterprises move Gen AI from pilot to production quickly, with pre-built accelerators, compliance guardrails, and real-world deployment architecture already in place.
Keep a tab on the GoML blog to stay tuned with the latest in AI and ML engineering.
Source: arXiv:2606.30534
FAQs
How does Orca's approach differ practically from training three separate models for text, image, and action?
Instead of three models with no shared understanding, Orca trains one shared representation (the World Latent) and attaches small, task-specific decoders on top. This means adding a new output type is cheaper than training a new foundation model from scratch, you only need to train a lightweight decoder, not redo the whole pretraining process.
Is the PRICE-V0.1 benchmark result independently verified?
No, it's a benchmark introduced by the same team in the same paper. That doesn't make the result invalid, but it means outside researchers haven't yet tested other models against it independently, so the comparison should be treated as preliminary.
Does matching an action-labeled robotics policy mean Orca is ready for real-world deployment?
Not yet, by the authors' own account. The robotic evaluations covered five relatively simple dual-arm tasks. Matching a labeled policy on simple tasks is a meaningful proof of concept, not evidence of readiness for complex, long-horizon manipulation work.
