A central problem in robotics is constructing insurance policies that generalize past the demonstrations they’re educated on. A coverage that succeeds in a coaching scene usually fails when object shapes, positions, or lighting change. Generalizing to those new situations requires the coverage to grasp the duties underlying physics, not simply mimic the demonstrations. This potential comes from the spine it’s constructed on.
The usual solution to construct a language-conditioned robotic coverage is so as to add an motion module to a pretrained vision-language mannequin (VLM), producing a vision-language-action (VLA) mannequin. This strategy has carried generalist manipulation a good distance. However a VLM spine learns to explain the world, not predict the way it evolves. That lacking dynamics mannequin is precisely what a robotic wants when a activity depends upon anticipating how a scene will change. A rising line of analysis replaces the language spine with a video world mannequin, producing world motion mannequin (WAM). NVIDIA researcher Jim Fan explored this shift in his Robotics’ Finish Sport discuss—an concept later summarized as “VLAs are useless, lengthy dwell World Motion Fashions.”
This submit explores how post-training can flip WAMs into specialised robotic coverage, how WAMs examine to VLAs, and why the open NVIDIA Cosmos 3 world mannequin offers a robust basis for constructing WAMs.
How post-trained insurance policies are constructed at the moment
Within the VLA paradigm, a pretrained VLM offers semantic understanding of scenes and language directions, whereas post-training learns to map that understanding to robotic actions. Fashionable generalist robotic insurance policies more and more construct on this strategy.
Nevertheless, a VLM is optimized to provide textual content about pictures, to not mannequin how a scene will evolve. It doesn’t be taught what occurs to a mug when the gripper closes, how a towel folds, the place an object lands when launched. VLAs generalize nicely semantically however are much less efficient at bodily generalization to unseen behaviors and environments, as their backbones mannequin language and notion quite than world dynamics.
What a WAM modifications
A WAM overcomes that dynamics-modeling restrict by constructing the coverage on a video world mannequin. As a result of the spine fashions how the world evolves, post-training doesn’t have to show dynamics from scratch, it specializes a mannequin that already has a physics prior. The NVIDIA analysis paper World Motion Fashions are Zero-shot Insurance policies exhibits that collectively predicting video and motion provides a coverage properties a VLA can’t simply purchase:
It learns from numerous knowledge. A VLA learns to map directions to trajectories, and infrequently requires near-identical demonstrations of the identical activity. A WAM learns physics: how objects transfer when pushed, grasped, or dropped. Any interplay knowledge teaches it one thing. A diversified dataset that may be wasted on a VLA turns into a coaching sign, and knowledge assortment will get cheaper.
It generalizes within the open world. Physics is extra common than semantics. The best way an object falls or slides doesn’t change with the article, so a mannequin that has realized dynamics carries that data into scenes and motions it was by no means educated on.
It adapts to new robots with few demonstrations. A mannequin that already understands bodily interplay wants far much less task-specific knowledge to specialize to a brand new arm or gripper.
For a group constructing a coverage, these are the sensible wins: much less knowledge to succeed in a given functionality, higher habits exterior the coaching distribution, and a shorter path to a brand new embodiment. They’re properties of the pretraining spine, in order that they present up in each coverage post-trained from it.
Cosmos 3 is a robust WAM basis
Cosmos 3 is an omni-model world basis mannequin constructed on a Combination-of-Transformers (MoT) structure. Multimodal enter flows by way of an autoregressive transformer for reasoning producing discrete tokens corresponding to textual content. This guides a diffusion transformer for steady modalities, together with picture, video, audio, and motion. Textual content is generated by next-token decoding; every thing else, together with actions, is synthesized by way of iterative denoising. A single mannequin spans these modalities whereas retaining the era mechanism finest suited to every. Cosmos 3 is available in three sizes: 4B NVIDIA Cosmos Edge, 16B NVIDIA Cosmos Nano, and 64B NVIDIA Cosmos 3 Tremendous.
What makes Cosmos 3 a robust basis for post-training is the breadth of its physical-world knowledge. The dataset consists of roughly 767M pictures,348M movies of real-world dynamics, 8M motion samples spanning robotic manipulation, autonomous driving, digicam movement, and selfish movement.
From world mannequin to robotic coverage: Cosmos 3 Coverage DROID fashions
Cosmos 3 is the start line for specialization. Cosmos3-Nano-Coverage-DROID is a 16B-parameter coverage post-trained from Cosmos 3 Nano for the DROID platform, which is a Franka Panda arm with a Robotiq gripper.A 4B model, Cosmos3-Edge-Coverage-DROID, is post-trained the identical manner, which can be utilized for on-device deployment. Given a language instruction and multi-camera observations, it generates robotic motion trajectories. Three properties observe from its omni basis:
It imagines whereas it acts. When the mannequin outputs actions, it could possibly additionally output a video: what the robotic’s cameras will see if these actions are executed. The motion and the anticipated consequence come from the identical mannequin, on the similar time.
It retains the complete omni structure. Submit-training removes nothing. The coverage checkpoint can nonetheless cause and generate video, not simply output joint positions.
The prior is measurable. The Cosmos 3 technical report compares two DROID insurance policies educated with the identical recipe, knowledge, and compute. One began from the bottom checkpoint, whereas the opposite began from an omni checkpoint educated on multi-domain motion knowledge. The omni checkpoint raised RoboLab success from 28.1% to 36.8%. That is clear proof that the development comes from the structure, not simply from scale.
Deployment issues
A WAM carries the complete generative world mannequin, not simply an motion head. It’s bigger than compact VLAs, however Cosmos 3 map to totally different deployment tiers quite than forcing one trade-off:
Workstation serving (Nano, 16B ). Cosmos3-Nano-Coverage runs beside the robotic quite than on board. Actual-world DROID deployment serves it on a single NVIDIA RTX PRO 6000, with the robotic streaming observations over the community and receiving motion chunks again.
On-device (Edge, 4B). Cosmos 3 Edge runs the identical coverage workload immediately on embedded {hardware}. It operates at robot-control decision (640×360 observations) and generates 32 actions per inference on NVIDIA Jetson Thor whereas attaining real-time management at 15 Hz. It’s supported throughout NVIDIA edge computer systems together with RTX PRO GPUs, DGX, GeForce RTX GPUs, and Jetson, together with the brand new Jetson T2000 and T3000 modules.
Why construct robotic insurance policies with Cosmos 3?
WAMs symbolize a shift from studying to behave to studying how the world evolves. Cosmos 3 makes this strategy sensible:
Open basis, SOTA place to begin. The bottom mannequin, datasets, post-training recipe, educated weights, analysis instruments, and serving stack are all launched below a license that allows business use.
Sooner adaptation. Sturdy bodily priors minimize the task-specific knowledge a brand new coverage wants. Convert your knowledge to the LeRobotDataset format the robot-learning ecosystem already data into, run the revealed recipe.
One basis, many robots. Every new embodiment, corresponding to Franka, dual-arm setups, UR, WidowX, nonetheless requires its personal post-training run. However all of them begin from the identical pretrained basis quite than pretraining a world mannequin from scratch, lowering the demonstrations wanted for every.
Get began
The quickest solution to consider whether or not a WAM beats your present VLA is to post-train one from Cosmos 3 by yourself knowledge and examine.
