Autonomous automobile (AV) improvement typically depends on separate fashions for trajectory era, high-level intent prediction, scene understanding, and knowledge labeling. This separation makes it laborious to match associated outputs, examine mannequin conduct, and reuse the identical representations throughout the event workflow.\u00a0<\/p>\n
NVIDIA Alpamayo 2 Tremendous is an open 34-billion-parameter reasoning vision-language-action (VLA) mannequin designed to speed up autonomous automobile (AV)\u00a0 improvement. It combines the 32-billion-parameter NVIDIA Cosmos 3 Tremendous Reasoner with a 2-billion-parameter diffusion-based Motion Skilled and is post-trained with reinforcement studying. The reasoner interprets multi-camera video, language context, and prior movement historical past, whereas the Motion Skilled converts the mannequin\u2019s ensuing inside illustration right into a future ego-vehicle trajectory.<\/p>\n
Alpamayo 2 Tremendous\u2019s notion expands to 360-degree protection throughout as much as seven cameras and may return a number of complementary outputs: future trajectories, Chain-of-Causation (CoC) reasoning traces, high-level meta-actions, grounded solutions to questions in regards to the scene, and reasoning auto-labels.<\/p>\n
This multi-task design offers AV builders a typical basis throughout a number of phases of the event workflow. The identical basis mannequin can be utilized as an offline coverage instructor, an analysis critic, an information engine, or a place to begin for brand new job customization, as a substitute of sustaining a separate mannequin for every stage of the workflow.<\/p>\n
This submit offers a hands-on introduction to 4 Alpamayo 2 Tremendous-enabled workflows:<\/p>\n
Generate trajectories and CoC reasoning traces, evaluating the leads to open-loop and closed-loop benchmarks.<\/p>\n
Predict meta-actions corresponding to yield, change lanes, and cease alongside a trajectory.<\/p>\n
Ask natural-language questions on a multi-camera driving scene.<\/p>\n
Generate CoC auto-labels with 2D grounding by yourself clips.<\/p>\n
The mannequin weights can be found on Hugging Face and the inference notebooks on GitHub. The mannequin is launched below OpenMDW-1.1, the Linux Basis permissive license for open mannequin distributions, which covers fine-tuning, spinoff fashions, and business redistribution. Distilled fashions could be deployed commercially with out additional permission from NVIDIA, and mannequin outputs carry no license situations.<\/p>\n
![\"Architecture](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image6-9.webp\")
Planning and reasoning<\/h2>\n
Reasoning by new situations is a basic drawback in autonomous driving. Navigating development zones, partially occluded pedestrians, uncommon right-of-way interactions, and objects getting into the roadway requires greater than matching a typical trajectory sample. A helpful driving mannequin should establish scene context that issues, join it to the suitable driving resolution, and produce an motion per that call.<\/p>\n
Trajectories and CoC traces: What and why<\/h3>\n
Alpamayo 2 Tremendous, like its predecessors, collectively produces output trajectories and CoC reasoning traces. The trajectory expresses what the ego automobile might do subsequent. The CoC reasoning hint offers insights into why a driving resolution was constituted of noticed scene context. Returning each outputs makes it simpler to grasp the mannequin\u2019s decision-making, curate troublesome instances, examine a deployed coverage with a bigger instructor, and diagnose whether or not a failure originated in notion, reasoning, or motion era. CoC traces additionally feed into the NVIDIA Halos security validation workflows by enabling introspection into the mannequin\u2019s understanding of the scene.\u00a0<\/p>\n
The Alpamayo 2 Tremendous repository\u2019s inference pocket book masses a surround-view clip, prepares ego-motion historical past, and samples a trajectory with its related CoC hint. The core inference step and outputs are proven beneath.<\/p>\n
\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
\nfrom alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
\nfrom alpamayo2_super.visualization import plot_inference_result<\/p>\n
knowledge = load_physical_aiavdataset(
\n “030c760c-ae38-49aa-9ad8-f5650a545d26”,
\n t0_us=2000000,
\n)<\/p>\n
mannequin = Alpamayo2Super.from_pretrained(“nvidia\/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″)
\nmodel_inputs = helper.prepare_model_inputs(knowledge, mannequin.config, mannequin.tokenizer)
\nmodel_inputs = helper.to_device(model_inputs, “cuda”)<\/p>\n
torch.cuda.manual_seed_all(42)
\nwith torch.autocast(“cuda”, dtype=torch.bfloat16):
\n pred_xyz, pred_rot, logprob, additional = mannequin.sample_trajectories_from_data(
\n knowledge=model_inputs,
\n top_p=0.98,
\n temperature=0.6,
\n num_traj_samples=1,
\n diffusion_kwargs={“inference_step”: 10},
\n return_extra=True,
\n )<\/p>\n
fig, metadata = plot_inference_result(
\n knowledge=knowledge,
\n pred_xyz=pred_xyz,
\n additional=additional,
\n)\n<\/p><\/div>\n
![\"Six](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image7-8.webp\")
Analysis strategies<\/h2>\n
To judge the mannequin\u2019s output reasoning and trajectory high quality, we will use open-loop and closed-loop analysis strategies. Open-loop analysis measures trajectory and reasoning high quality on recorded scenes by evaluating them to ground-truth labels. <\/p>\n
Alpamayo 2 Tremendous achieves the next outcomes:<\/p>\n
Trajectory prediction: Throughout 1,434 difficult samples from the Bodily AI AV Dataset, it data a 6.4-second minADE_6 of 0.911 m.<\/p>\n
AV reasoning: It scores 0.433 on the Bodily AI AV Reasoning Benchmark.<\/p>\n
LingoQA: Alpamayo 2 Tremendous achieves 79.2 on the LingoQA benchmark, rating first amongst 37 evaluated fashions. With 34 billion parameters, it leads Qwen2.5-VL (72B) by 17.0 factors, Qwen3-VL (32B) by 7.0 factors, Gemini 2.5 Professional by 15.1 factors, and GPT-4o by 23.2 factors.<\/p>\n
Decrease minADE_6 values point out higher trajectory predictions; larger reasoning scores point out higher efficiency.<\/p>\n
The principle problem with open-loop metrics, nonetheless, is that they consider predictions towards a set, prerecorded future and subsequently don\u2019t seize what occurs after the mannequin\u2019s first motion, which can have an effect on the remainder of the scene.\u00a0<\/p>\n
For instance, if the ego automobile adjustments lanes, an open-loop replay could proceed shifting an adjoining automobile alongside its recorded trajectory fairly than accounting for the way it might reply to the ego automobile.\u00a0<\/p>\n
Closed-loop simulation executes every predicted motion throughout the scene and, when the simulator contains reactive conduct fashions, captures how surrounding brokers could react. NVIDIA AlpaSim permits this by repeatedly rendering observations, querying the coverage, and making use of its actions so builders can measure these closed-loop results over time.<\/p>\n
To run Alpamayo 2 Tremendous on an AlpaSim analysis suite, use the corresponding shell command:<\/p>\n
\nwizard.log_dir=$PWD\/tutorial eval.video.video_layouts=[REASONING_OVERLAY]\n<\/div>\n
with the next AlpaSim wizard configuration:<\/p>\n
\n# – \/driver: alpamayo2
\n# Sort validation occurs at driver runtime through OmegaConf.structured merge<\/p>\n
defaults:
\n – alpamayo_configs # Digicam and simulation configs for 4-cam 10Hz
\n – _self_ # YAML values override schema defaults<\/p>\n
# Alpamayo 2 Tremendous Driver Configuration for Alpasim<\/p>\n
# Logging stage (makes use of wizard’s world setting)
\nlog_level: ${wizard.log_level}<\/p>\n
# Mannequin configuration
\nmannequin:
\n model_type: alpamayo2 # Entry-point identify in alpasim.fashions registry
\n # HuggingFace mannequin ID (requires cached obtain or hf authentication within the driver container)
\n checkpoint_path: “nvidia\/Alpamayo2-Tremendous”
\n # # Various native path to a pre-downloaded mannequin
\n # checkpoint_path: “\/mnt\/drivers\/alpamayo2\/Alpamayo2-Tremendous”
\n machine: “cuda”
\n # Allow classifier-free steering navigation sampling (NOTE: this requires 2 GPUs with at the very least 70 GB VRAM).
\n # Set to true solely when enough GPU reminiscence is out there.
\n use_classifier_free_guidance_nav: false<\/p>\n
# Server configuration
\nhost: “0.0.0.0”
\nport: ???<\/p>\n
# Inference configuration
\ninference:
\n use_cameras:
\n – camera_cross_left_120fov
\n – camera_front_wide_120fov
\n – camera_cross_right_120fov
\n – camera_front_tele_30fov
\n max_batch_size: 1 # A2Super is reminiscence intensive, begin with batch dimension 1
\n subsample_factor: 1
\n context_length: 4 # A2Super makes use of 4 temporal frames per digital camera<\/p>\n
# Route configuration \u2014 A2Super makes use of language-only navigation, not waypoint instructions
\nroute:
\n use_waypoint_commands: false<\/p>\n
# Output configuration
\noutput_dir: “\/mnt\/output\/driver”<\/p>\n
# Trajectory optimization (disabled by default for A2Super)
\ntrajectory_optimizer:
\n enabled: false<\/p>\n
plot_debug_images: false\n<\/p><\/div>\n On 913 reconstructed scenes, Alpamayo 2 Tremendous obtains an AlpaSim Rating of 1.50 \u00b1 0.13. This closed-loop rating enhances open-loop outcomes by revealing collisions, highway departures, shut encounters, and different failures that may emerge solely after the coverage influences future observations.<\/p>\n A trajectory is exact, nevertheless it doesn\u2019t all the time present a compact description of intent. Meta-actions summarize a plan by way of high-level choices corresponding to yield, change lanes, cease, or speed up. These outputs may help bridge an end-to-end basis mannequin and a modular AV stack: a downstream planner can eat the choice, an evaluator can verify whether or not trajectory geometry agrees with intent, and groups can search their knowledge corpus for specific maneuvers.<\/p>\n The meta-actions pocket book exhibits how one can produce meta-action outputs with an instance scene. For an in depth record of supported meta-actions, refer to those lists. The core inference step and outputs are proven beneath.<\/p>\n knowledge = load_physical_aiavdataset( mannequin = Alpamayo2Super.from_pretrained(“nvidia\/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″) torch.cuda.manual_seed_all(42) cot = outcome[“cot”][0] To judge meta-action accuracy, we examine Alpamayo 2 Tremendous\u2019s output with ground-truth labels and report the ensuing classification accuracy by way of intersection-over-union (IoU) for the three elements of its meta-action taxonomy: lateral, longitudinal, and lane-wise.<\/p>\n On an inside set of 94K clips with ground-truth meta-action knowledge, Alpamayo 2 Tremendous achieves 74.59 lateral IoU, 61.91 longitudinal IoU, and 73.55 lane-wise IoU throughout its meta-action taxonomy.\u00a0<\/p>\n Planning is just one method to make use of a driving basis mannequin. Visible query answering (VQA) exposes the mannequin\u2019s scene understanding instantly by pure language. Builders can ask about key components within the scene, how they have an effect on driving conduct, why the ego automobile ought to gradual, or about different elements of situations.ne-change hole.<\/p>\n VQA is efficacious for interactive debugging and knowledge operations. It could actually assist a developer examine why a coverage behaved a sure method, construct semantic filters over giant clip collections, and generate candidate annotations for human evaluation. With surround-view enter, questions can reference aspect and rear context {that a} front-view-only mannequin would miss.<\/p>\n For every clip, Alpamayo 2 Tremendous can generate solutions and spatially localize referenced actors by predicting 2D bounding bins within the related digital camera frames. This grounding makes outputs extra helpful than free-form textual content alone. Reviewers can confirm which particular object the mannequin is referring to, automated checks can flag lacking or inconsistent bins, and downstream fashions can leverage (e.g., by distillation) these tighter hyperlinks between visible proof, reasoning, and motion.<\/p>\n The scene understanding and VQA pocket book exhibits how one can carry out VQA with the identical inputs. The core inference step and outputs are proven beneath.<\/p>\n knowledge = load_physical_aiavdataset( mannequin = Alpamayo2Super.from_pretrained(“nvidia\/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″) The mannequin can return two sorts of output for a similar enter: Determine 4, above, exhibits a natural-language description of the scene, whereas Determine 5, beneath, exhibits the identical mannequin spatially localizing referenced objects by predicting 2D bounding bins within the related digital camera body.<\/p>\n To judge VQA efficiency, we examine Alpamayo 2 Tremendous\u2019s generated responses with floor fact solutions throughout an inside set of 8K question-answer pairs. The mannequin achieves 0.652 reply similarity (larger is best), in contrast with Qwen3-VL 32B at 0.450. For 2D grounding, we examine the anticipated bounding bins to ground-truth annotations utilizing IoU and procure 0.71 in comparison with 0.17 for Qwen3-VL 32B. Collectively, these measurements consider whether or not the mannequin each solutions questions precisely and associates its responses with the proper visible proof throughout the surround-view cameras.<\/p>\n Reasoning fashions want decision-grounded reasoning knowledge, however labeling long-tail driving clips by hand is dear and gradual. Annotators should examine temporal and multi-camera context, establish the causal actors, describe how they have an effect on the ego automobile, and maintain the label per the supposed maneuver. Alpamayo 2 Tremendous can function an offline auto-labeler that proposes this construction at scale. This could compress annotation cycles from months to days.<\/p>\n The CoC auto-labeling pocket book accepts clips within the launched schema and writes one structured file per chosen keyframe.<\/p>\n knowledge = load_physical_aiavdataset( mannequin = Alpamayo2Super.from_pretrained(“nvidia\/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″)<\/p>\n task_inputs = prepare_text_generation_inputs( torch.cuda.manual_seed_all(42) auto_labeling_text = outcome[“cot_auto_labeling”][0] By default, Alpamayo 2 Tremendous assumes entry to future ego-trajectory info. Nonetheless, it may additionally auto-label knowledge that doesn’t include future trajectory info:<\/p>\n # Get the mannequin to foretell a future trajectory of its personal, and use that in auto-labeling because the “noticed” future movement. torch.cuda.manual_seed_all(42) task_inputs = prepare_text_generation_inputs( torch.cuda.manual_seed_all(42) auto_labeling_text = outcome[“cot_auto_labeling”][0]Generate meta-actions<\/h3>\n
\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
\nfrom alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
\nfrom alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs<\/p>\n
\n “030c760c-ae38-49aa-9ad8-f5650a545d26”,
\n t0_us=2000000,
\n)<\/p>\n
\ntask_inputs = prepare_text_generation_inputs(
\n knowledge=knowledge,
\n model_config=mannequin.config,
\n tokenizer=mannequin.tokenizer,
\n job=”meta_action”,
\n)
\ntask_inputs = helper.to_device(task_inputs, “cuda”)<\/p>\n
\nwith torch.autocast(“cuda”, dtype=torch.bfloat16):
\n outcome = generate_text(
\n mannequin,
\n task_inputs,
\n top_p=0.98,
\n temperature=0.6,
\n max_new_tokens=512,
\n )<\/p>\n
\nmeta_action = outcome[“meta_action”][0]
\nprint(“Chain-of-Causation:n”, cot)
\nprint(“nMeta-action:n”, meta_action)\n<\/p><\/div>\nConsider meta-action accuracy<\/h3>\n
![\"Camera](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image5-14.webp\")
Scene understanding<\/h2>\n
Querying multi-camera scenes with VQA and 2D grounding<\/h3>\n
\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
\nfrom alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
\nfrom alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs<\/p>\n
\n “ea4a6729-bf33-4997-905b-cd58774a3580”,
\n t0_us=7500000,
\n)<\/p>\n
\ntask_inputs = prepare_vqa_inputs(
\n knowledge=knowledge,
\n model_config=mannequin.config,
\n tokenizer=mannequin.tokenizer,
\n query=”Describe the driving scene and establish the important thing site visitors components that ought to affect ego conduct.”,
\n)
\ntask_inputs = helper.to_device(task_inputs, “cuda”)
\nwith torch.autocast(“cuda”, dtype=torch.bfloat16):
\n outcome = generate_text(
\n mannequin,
\n task_inputs,
\n top_p=1.0,
\n temperature=0.1,
\n max_new_tokens=1024,
\n )
\nreply = outcome[“answer”][0]
\nprint(reply)\n<\/p><\/div>\n![\"Six](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image9-3.webp\")
![\"A](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image1-14.webp\")
Consider VQA response and grounding high quality<\/h3>\n
Auto-labeling<\/h2>\n
Generate structured CoC auto-labels<\/h3>\n
\nfrom alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
\nfrom alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
\nfrom alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs<\/p>\n
\n “b5f3756c-4f0e-4298-a1ff-cc92ed392ae0”,
\n t0_us=11000000,
\n)<\/p>\n
\n knowledge=knowledge,
\n model_config=mannequin.config,
\n tokenizer=mannequin.tokenizer,
\n job=”auto_labeling”,
\n)
\ntask_inputs = helper.to_device(task_inputs, “cuda”)<\/p>\n
\nwith torch.autocast(“cuda”, dtype=torch.bfloat16):
\n outcome = generate_text(
\n mannequin,
\n task_inputs,
\n top_p=0.98,
\n temperature=0.6,
\n max_new_tokens=1024,
\n )<\/p>\n
\nauto_labeling_json = outcome[“cot_auto_labeling_json”][0]
\nprint(json.dumps(auto_labeling_json, indent=2))\n<\/p><\/div>\n
\ntrajectory_inputs = helper.prepare_model_inputs(knowledge, mannequin.config, mannequin.tokenizer)
\ntrajectory_inputs = helper.to_device(trajectory_inputs, “cuda”)<\/p>\n
\nwith torch.autocast(“cuda”, dtype=torch.bfloat16):
\n pred_xyz, pred_rot, _, additional = mannequin.sample_trajectories_from_data(
\n knowledge=trajectory_inputs,
\n top_p=0.98,
\n temperature=0.6,
\n num_traj_samples=1,
\n diffusion_kwargs={“inference_step”: 10},
\n return_extra=True,
\n )
\n future_xyz = pred_xyz[:, 0, 0].detach().cpu()
\n future_rot = pred_rot[:, 0, 0].detach().cpu()
\n # In case you wish to see the mannequin’s reasoning, uncomment these:
\n # trajectory_cot = str(additional[“cot”].reshape(-1)[0])
\n # print(“trajectory_cot:n”, trajectory_cot)<\/p>\n
\n knowledge=knowledge,
\n model_config=mannequin.config,
\n tokenizer=mannequin.tokenizer,
\n job=”auto_labeling”,
\n # That is the place the anticipated trajectories are handed in:
\n future_xyz=future_xyz,
\n future_rot=future_rot,
\n)
\ntask_inputs = helper.to_device(task_inputs, “cuda”)<\/p>\n
\nwith torch.autocast(“cuda”, dtype=torch.bfloat16):
\n outcome = generate_text(
\n mannequin,
\n task_inputs,
\n top_p=0.98,
\n temperature=0.6,
\n max_new_tokens=1024,
\n )<\/p>\n
\nauto_labeling_json = outcome[“cot_auto_labeling_json”][0]
\nprint(json.dumps(auto_labeling_json, indent=2))\n<\/p><\/div>\n
![\"Multi-camera](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/07\/image7.gif\")