Bodily AI programs should perceive the actual world earlier than they will act inside it. Robots, autonomous autos, and sensible areas want to grasp what’s taking place of their world, predict what’s prone to occur subsequent, and generate actions for particular environments, embodiments, and duties.
NVIDIA Cosmos 3 is a frontier basis mannequin for bodily AI that mixes bodily reasoning, world era, and motion era inside a single open mannequin.
NVIDIA is open sourcing Cosmos 3 fashions, coaching scripts, deployment instruments, and datasets to make bodily AI improvement extra open and reproducible. This weblog submit covers the basics of Cosmos 3, highlights key ideas from the technical report, guides by means of technical workflows, and exhibits how groups constructing robotic manipulation programs, autonomous autos, and warehouse monitoring options can get began.
Key highlights of this launch embody:
NVIDIA Cosmos 3 Nano and NVIDIA Cosmos 3 Tremendous mannequin checkpoints on Hugging Face with code on GitHub.
Open datasets for bodily AI purposes like robotics and autonomous driving.
Open post-training scripts for adapting Cosmos 3 to your area.
Cosmos NIM microservices for simple, optimized deployment on NVIDIA GPUs.
What’s new in Cosmos 3
Earlier Cosmos releases separated world era, bodily understanding, and managed scene era into completely different fashions and workflows. This launch unifies these capabilities with a Combination-of-Transformers (MoT) structure constructed round two towers.
Reasoner tower: A vision-language mannequin (VLM) that interprets multimodal observations like photographs, movies, and textual content. This tower makes use of an autoregressive structure to interpret the enter and perceive movement, object interactions, and different bodily context. This serves because the ‘mind’ that causes concerning the world earlier than any era occurs.
Generator tower: Generates future observations and motion sequences. This tower makes use of a diffusion-based course of to generate physics-aware video and motion outputs which might be conditioned on the reasoner tower’s understanding. The reasoner might be known as independently, however the generator at all times prompts each towers for guided era.
This structure allows a single mannequin to do reasoning and era duties, simplifying improvement by eliminating orchestration between a number of fashions and inference pipelines.
Select the proper mannequin dimension
Two Cosmos 3 fashions are at present out there:
Cosmos 3 Nano is the compact model with 16B parameters and optimized for environment friendly inference. It’s designed to run on workstation-grade compute, just like the NVIDIA RTX PRO 6000 GPU for real-time robotics inference and bodily AI purposes.
Cosmos 3 Tremendous is a 64B parameter mannequin designed for max high quality and functionality. It delivers the best benchmark scores and targets datacenter deployment on NVIDIA Hopper and NVIDIA Blackwell GPUs, making it appropriate for large-scale artificial information era and superior bodily reasoning workloads.
Supported modalities
Cosmos 3 helps the next enter and output modalities by means of its unified structure:
Open datasets for bodily AI
With the Cosmos 3 launch, NVIDIA is open-sourcing six artificial information era (SDG) datasets on Hugging Face. These cowl robotics, physics simulation, spatial reasoning, human movement, driving, and warehouse environments, and can be utilized for post-training Cosmos 3 and different fashions:
Bodily AI World Mannequin Artificial Datasets embody:
NVIDIA Cosmos Human Analysis benchmark
The NVIDIA Cosmos Human Analysis (HUE) framework assesses Cosmos 3 generator high quality throughout consultant area duties.
As SOTA video era fashions saturate present automated leaderboards, rating variations between releases are sometimes too slender for significant comparability. HUE shifts analysis from subjective grading to goal truth verification, enabling fine-grained comparability between top-tier fashions. The result’s a extra dependable high quality sign for each fast iteration and rigorous launch choices backed by full human analysis.
HUE evaluates video era high quality utilizing atomic binary verification. Every generated video is decomposed into single-fact sure/no questions throughout 4 dimensions—semantic alignment, bodily legal guidelines, geometric reasoning, and visible integrity—spanning seven Bodily AI domains, together with robotics, autonomous autos, and physics. These questions are generated by a VLM pipeline, refined by human consultants, and launched as open supply on Hugging Face.
Benchmark outcomes
Cosmos 3 has been evaluated throughout a number of benchmark suites masking bodily AI reasoning, era high quality, and domain-specific efficiency.
Reasoning benchmarks
Cosmos 3 Tremendous and Cosmos 3 Nano lead on VANTAGE-Bench on the 32B tier and the 8B tier, respectively:
VANTAGE-Bench: First public benchmark for evaluating vision-language fashions on real-world fixed-camera footage throughout warehouses, transportation, and sensible areas.
Site visitors Anomaly Reasoning (TAR): A brand new leaderboard for detecting and reasoning anomalous occasions in transportation footage and the official leaderboard for AI Metropolis Problem 2026 Observe 3.
Generator benchmarks
Cosmos 3 is the open-source SOTA and at present leads on PAI-Bench, R-Bench Physics-IQ, and RoboLab throughout public leaderboards:
Synthetic Evaluation: A benchmarking platform that ranks AI fashions for textual content, picture, and video era. Cosmos 3 is the main open supply mannequin on the Textual content to Picture leaderboard and Picture to Video (no audio) leaderboard.
R-Bench: A benchmark for evaluating video-based world fashions in robotic video era. It assesses job completion and visible high quality by means of sub-metrics like structural consistency, bodily plausibility, and execution completeness.
PAI-Bench: A unified benchmark evaluating bodily AI throughout video understanding and video era, spanning domains like robotics, autonomous autos, and physics widespread sense.
Physics-IQ: A benchmark of real-world movies that checks whether or not generative video fashions really perceive bodily rules, moderately than simply reaching visible realism.
RoboLab: A simulation benchmark for evaluating task-generalist robotic insurance policies.
Coaching recipes
A central part of the Cosmos 3 launch is a completely open set of coaching recipes. Past mannequin checkpoints, this launch gives code, configs, and workflows for adapting Cosmos 3 to new domains, embodiments, and datasets.
Supervised Tremendous-Tuning post-training
Supervised Tremendous-Tuning (SFT) allows builders to adapt a Cosmos 3 mannequin to their very own information. The launched recipes embody imaginative and prescient era post-training for customized video datasets, in addition to action-oriented recipes for robotics and bodily AI workflows. Builders can customise Cosmos 3 for his or her goal domains throughout robotics, autonomous driving, and warehouse automation.
The post-training code and configs can be found on GitHub.
Motion post-training
Motion post-training adapts Cosmos 3 for action-aware Bodily AI purposes, together with ahead dynamics, inverse dynamics, and coverage era. Builders can post-train Cosmos 3 on action-labeled information. For robotics purposes, this consists of a number of necessary workflows: producing future observations conditioned on robotic actions, inferring the actions behind noticed demonstrations, and predicting motion sequences from present observations and job prompts. This makes Cosmos 3 a powerful basis for world motion modeling and coverage studying.
Deploy with NVIDIA NIM Microservices
Cosmos 3 fashions are additionally out there as NVIDIA NIM microservices for optimized, production-ready deployment. NIM microservices bundle the mannequin with optimized inference runtimes, delivering excessive efficiency with out the necessity to manually tune serving infrastructure. NIM microservices are simpler to make use of for inference workflows in comparison with the Cosmos 3 repo on GitHub, which is most well-liked for post-training workflows.
The Cosmos 3 Reasoner NIM is on the market at present, delivering the reasoning capabilities of the Cosmos 3 mannequin. Hold posted for the Cosmos 3 Generator NIM, which gives full era capabilities of the Cosmos 3 mannequin.
Optimizations made to speed up inference
Quantization: Cosmos 3 NIM helps choosing BF16, FP8, or NVFP4 quantized checkpoints. The NVFP4 quantization reduces the mannequin’s numerical precision from BF16 to 4-bit floating level, reaching as much as 2x inference speedup.
vLLM: Is an open supply inference engine that makes use of strategies like steady batching, paged consideration, and tensor parallelism to serve LLMs effectively. The Cosmos 3 Reasoner NIM serving stack is constructed on vLLM for increased throughput in comparison with standard serving approaches. Cosmos 3 Nano is able to run with vLLM-omni and NVIDIA Dynamo for prime efficiency.
Environment friendly Video Sampling (EVS): This system reduces the variety of video tokens fed into the VLM throughout inference, rushing up the Cosmos Purpose NIM. EVS works on the chunk stage, conserving essentially the most distinctive chunks of every body and pruning the remainder. Smaller GPUs have a tendency to learn extra from this system.
Easy methods to run the NIM
An NVIDIA NGC API secret is required to drag the containers and obtain the Cosmos 3 fashions from NGC.
To drag and run the Cosmos 3 Nano Reasoner NIM. For the Cosmos 3 Tremendous Reasoner NIM, specify NIM_MODEL_SIZE=tremendous.
-e NGC_API_KEY=$NGC_API_KEY
-e NIM_MODEL_SIZE=nano
-p 8000:8000
nvcr.io/nim/nvidia/cosmos3-reasoner:newest
Discover particulars on API utilization and extra within the documentation.
Get began
Acknowledgments
Cosmos 3 is the results of wonderful collaboration between many groups and folks throughout NVIDIA, together with Aditi, Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, Aarti Basant, Mukesh Beladiya, Mohammad Qazim Bhat, Zaid Pervaiz Bhat, Dan Blick, Vanni Brighella, Han Cai, Tiffany Cai, Eric Cameracci, Jiaxin Cao, Yulong Cao, Mark Carlson, Carlos Casanova, Ting-Yun Chang, Yan Chang, Yu-Wei Chao, Prithvijit Chattopadhyay, Roshan Chaudhari, Chieh-Yun Chen, Junyu Chen, Ke Chen, Qizhi Chen, Wenkai Chen, Xiaotong Chen, Yu Chen, An-Chieh Cheng, Click on Cheng, Xiu Chia, Jeana Choi, Chaeyeon Chung, Wenyan Cong, Yin Cui, Magdalena Dadela, Nalin Dadhich, Wenliang Dai, Joyjit Daw, Alperen Degirmenci, Rodrigo Vieira Del Monte, Robert Denomme, Sameer Dharur, Marco Di Lucca, Ke Ding, Wenhao Ding, Yifan Ding, Yuzhu Dong, Nicole Drumheller, Yilun Du, Aigul Dzhumamuratova, Aleksandr Efitorov, Hamid Eghbalzadeh, Naomi Eigbe, Imad El Hanafi, Hassan Eslami, Benedikt Falk, Jiaojiao Fan, Jim Fan, Amol Fasale, Sergiy Fefilatyev, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Vikram Fugro, Prashant Gaikwad, TJ Galda, Katelyn Gao, Yihuai Gao, Wenhang Ge, Sreyan Ghosh, Arushi Goel, Vivek Goel, Akash Gokul, Rama Govindaraju, Jinwei Gu, Miguel Guerrero, Elfie Guo, Aryaman Gupta, Siddharth Gururani, Hugo Hadfield, Tune Han, Ankur Handa, Zekun Hao, Mohammad Harrim, Ali Hassani, Nathan Hayes-Roth, Yufan He, Chris Helvig, Cyrus Hogg, Madison Huang, Michael Huang, Sophia Huang, Yufan Huang, Jacob Huffman, DeLesley Hutchins, Suneel Indupuru, Boris Ivanovic, Arihant Jain, Joel Jang, Ryan Ji, Yanan Jian, Dongfu Jiang, Jingyi Jin, Atharva Joshi, Nikhilesh Joshi, Pranjali Joshi, Jaehun Jung, Weiwei Kang, Scott Kassekert, Jan Kautz, Ashna Khetan, Julia Kiczka, Slawek Kierat, Gwanghyun Kim, Kuno Kim, Sunny Kim, Kezhi Kong, Xin Kong, Zhifeng Kong, Tomasz Kornuta, Egor Krivov, Hui Kuang, Saurav Kumar, Chia-Wen Kuo, George Kurian, Wojciech Kutak, JF Lafleche, Himangshu Lahkar, Omar Laymoun, Jayjun Lee, Sanggil Lee, Gabriele Leone, Boyi Li, Freya Li, Jiajun Li, Jinfeng Li, Ling Li, Pengcheng Li, Shangru Li, Tingle Li, Xiaolong Li, Xuan Li, Zhaoshuo Li, Zhiqi Li, Hao Liang, Maosheng Liao, Chen-Hsuan Lin, Tsung-Yi Lin, Ming-Yu Liu, Sifei Liu, Zihan Liu, Hai Loc Lu, Xiangyu Lu, Alice Luo, Ruipu Luo, Wenjie Luo, Jiangran Lyu, Martin Ding Ma, Nic Ma, Qianli Ma, Dawid Majchrowski, Louis Marcoux, Miguel Martin, Qing Miao, Ashkan Mirzaei, Shreyas Misra, Kaichun Mo, Durra Mohsin, Hyejin Moon, Pawel Morkisz, Saeid Motiian, Kirill Motkov, Seungjun Nah, Yashraj Narang, Deepak Narayanan, Thabang Ngazimbi, Julian Ouyang, David Web page, Yatian Pang, Sehwi Park, Mahesh Patekar, Mostofa Patwary, Marco Pavone, Trung Pham, Wei Ping, Soha Pouya, Shrimai Prabhumoye, Varun Praveen, Delin Qu, Hesam Rabeti, Morteza Ramezanali, Marilyn Reeb, Xuanchi Ren, Kristen Rumley, Wojciech Rymer, Jun Saito, Yeongho Seol, John Shao, Piyush Shekdar, Tianwei Shen, Humphrey Shi, Min Shi, Stella Shi, Kevin Shih, Mohammad Shoeybi, Mateusz Sieniawski, Shuran Tune, Alexander Sotelo, Amir Sotoodeh, Sunil Srinivasa, Vignesh Srinivasakumar, Bartosz Stefaniak, Rahul Heinrich Steiger, Shangkun Solar, Jiaxiang Tang, Shitao Tang, Yangyang Tang, Yue Tang, Tolou Tavakkoli, Kayley Ting, Krzysztof Tomala, Wei-Cheng Tseng, Jibin Varghese, Sergei Vasilev, Thomas Volk, Raju Wagwani, Roger Waleffe, Andrew Z. Wang, Boxiang Wang, Haoxiang Wang, Qiao Wang, Shihao Wang, Shijie Wang, Ting-Chun Wang, Yan Wang, Yu Wang, David Wehr, Fangyin Wei, Xinshuo Weng, Jay Zhangjie Wu, Kedi Wu, Hongchi Xia, Summer season Xiao, Tianjun Xiao, Kevin Xie, Daguang Xu, Jiashu Xu, Mengyao Xu, Ruqing Xu, Xingqian Xu, Yao Xu, Dinghao Yang, Dong Yang, Hans Yang, Xiaodong Yang, Xuning Yang, Yichu Yang, Yurong You, Zhiding Yu, Hao Yuan, Simon Yuen, Xiaohui Zeng, Pengcuo Zeren, Cindy Zha, Haotian Zhang, Jenny Zhang, Jing Zhang, Liangkai Zhang, Paris Zhang, Shun Zhang, Xuanmeng Zhang, Zhizheng Zhang, Ann Zhao, Yilin Zhao, Yuliya Zhautouskaya, Charles Zhou, Fengzhe Zhou, Shilin Zhu, Yuke Zhu, Dima Zhylko, and Artur Zolkowski.
