Autonomous automobile (AV) improvement typically depends on separate fashions for trajectory era, high-level intent prediction, scene understanding, and knowledge labeling. This separation makes it laborious to match associated outputs, examine mannequin conduct, and reuse the identical representations throughout the event workflow.
NVIDIA Alpamayo 2 Tremendous is an open 34-billion-parameter reasoning vision-language-action (VLA) mannequin designed to speed up autonomous automobile (AV) improvement. It combines the 32-billion-parameter NVIDIA Cosmos 3 Tremendous Reasoner with a 2-billion-parameter diffusion-based Motion Skilled and is post-trained with reinforcement studying. The reasoner interprets multi-camera video, language context, and prior movement historical past, whereas the Motion Skilled converts the mannequin’s ensuing inside illustration right into a future ego-vehicle trajectory.
Alpamayo 2 Tremendous’s notion expands to 360-degree protection throughout as much as seven cameras and may return a number of complementary outputs: future trajectories, Chain-of-Causation (CoC) reasoning traces, high-level meta-actions, grounded solutions to questions in regards to the scene, and reasoning auto-labels.
This multi-task design offers AV builders a typical basis throughout a number of phases of the event workflow. The identical basis mannequin can be utilized as an offline coverage instructor, an analysis critic, an information engine, or a place to begin for brand new job customization, as a substitute of sustaining a separate mannequin for every stage of the workflow.
This submit offers a hands-on introduction to 4 Alpamayo 2 Tremendous-enabled workflows:
Generate trajectories and CoC reasoning traces, evaluating the leads to open-loop and closed-loop benchmarks.
Predict meta-actions corresponding to yield, change lanes, and cease alongside a trajectory.
Ask natural-language questions on a multi-camera driving scene.
Generate CoC auto-labels with 2D grounding by yourself clips.
The mannequin weights can be found on Hugging Face and the inference notebooks on GitHub. The mannequin is launched below OpenMDW-1.1, the Linux Basis permissive license for open mannequin distributions, which covers fine-tuning, spinoff fashions, and business redistribution. Distilled fashions could be deployed commercially with out additional permission from NVIDIA, and mannequin outputs carry no license situations.
Planning and reasoning
Reasoning by new situations is a basic drawback in autonomous driving. Navigating development zones, partially occluded pedestrians, uncommon right-of-way interactions, and objects getting into the roadway requires greater than matching a typical trajectory sample. A helpful driving mannequin should establish scene context that issues, join it to the suitable driving resolution, and produce an motion per that call.
Trajectories and CoC traces: What and why
Alpamayo 2 Tremendous, like its predecessors, collectively produces output trajectories and CoC reasoning traces. The trajectory expresses what the ego automobile might do subsequent. The CoC reasoning hint offers insights into why a driving resolution was constituted of noticed scene context. Returning each outputs makes it simpler to grasp the mannequin’s decision-making, curate troublesome instances, examine a deployed coverage with a bigger instructor, and diagnose whether or not a failure originated in notion, reasoning, or motion era. CoC traces additionally feed into the NVIDIA Halos security validation workflows by enabling introspection into the mannequin’s understanding of the scene.
The Alpamayo 2 Tremendous repository’s inference pocket book masses a surround-view clip, prepares ego-motion historical past, and samples a trajectory with its related CoC hint. The core inference step and outputs are proven beneath.
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.visualization import plot_inference_result
knowledge = load_physical_aiavdataset(
“030c760c-ae38-49aa-9ad8-f5650a545d26”,
t0_us=2000000,
)
mannequin = Alpamayo2Super.from_pretrained(“nvidia/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″)
model_inputs = helper.prepare_model_inputs(knowledge, mannequin.config, mannequin.tokenizer)
model_inputs = helper.to_device(model_inputs, “cuda”)
torch.cuda.manual_seed_all(42)
with torch.autocast(“cuda”, dtype=torch.bfloat16):
pred_xyz, pred_rot, logprob, additional = mannequin.sample_trajectories_from_data(
knowledge=model_inputs,
top_p=0.98,
temperature=0.6,
num_traj_samples=1,
diffusion_kwargs={“inference_step”: 10},
return_extra=True,
)
fig, metadata = plot_inference_result(
knowledge=knowledge,
pred_xyz=pred_xyz,
additional=additional,
)
Analysis strategies
To judge the mannequin’s output reasoning and trajectory high quality, we will use open-loop and closed-loop analysis strategies. Open-loop analysis measures trajectory and reasoning high quality on recorded scenes by evaluating them to ground-truth labels.
Alpamayo 2 Tremendous achieves the next outcomes:
Trajectory prediction: Throughout 1,434 difficult samples from the Bodily AI AV Dataset, it data a 6.4-second minADE_6 of 0.911 m.
AV reasoning: It scores 0.433 on the Bodily AI AV Reasoning Benchmark.
LingoQA: Alpamayo 2 Tremendous achieves 79.2 on the LingoQA benchmark, rating first amongst 37 evaluated fashions. With 34 billion parameters, it leads Qwen2.5-VL (72B) by 17.0 factors, Qwen3-VL (32B) by 7.0 factors, Gemini 2.5 Professional by 15.1 factors, and GPT-4o by 23.2 factors.
Decrease minADE_6 values point out higher trajectory predictions; larger reasoning scores point out higher efficiency.
The principle problem with open-loop metrics, nonetheless, is that they consider predictions towards a set, prerecorded future and subsequently don’t seize what occurs after the mannequin’s first motion, which can have an effect on the remainder of the scene.
For instance, if the ego automobile adjustments lanes, an open-loop replay could proceed shifting an adjoining automobile alongside its recorded trajectory fairly than accounting for the way it might reply to the ego automobile.
Closed-loop simulation executes every predicted motion throughout the scene and, when the simulator contains reactive conduct fashions, captures how surrounding brokers could react. NVIDIA AlpaSim permits this by repeatedly rendering observations, querying the coverage, and making use of its actions so builders can measure these closed-loop results over time.
To run Alpamayo 2 Tremendous on an AlpaSim analysis suite, use the corresponding shell command:
wizard.log_dir=$PWD/tutorial eval.video.video_layouts=[REASONING_OVERLAY]
with the next AlpaSim wizard configuration:
# – /driver: alpamayo2
# Sort validation occurs at driver runtime through OmegaConf.structured merge
defaults:
– alpamayo_configs # Digicam and simulation configs for 4-cam 10Hz
– _self_ # YAML values override schema defaults
# Alpamayo 2 Tremendous Driver Configuration for Alpasim
# Logging stage (makes use of wizard’s world setting)
log_level: ${wizard.log_level}
# Mannequin configuration
mannequin:
model_type: alpamayo2 # Entry-point identify in alpasim.fashions registry
# HuggingFace mannequin ID (requires cached obtain or hf authentication within the driver container)
checkpoint_path: “nvidia/Alpamayo2-Tremendous”
# # Various native path to a pre-downloaded mannequin
# checkpoint_path: “/mnt/drivers/alpamayo2/Alpamayo2-Tremendous”
machine: “cuda”
# Allow classifier-free steering navigation sampling (NOTE: this requires 2 GPUs with at the very least 70 GB VRAM).
# Set to true solely when enough GPU reminiscence is out there.
use_classifier_free_guidance_nav: false
# Server configuration
host: “0.0.0.0”
port: ???
# Inference configuration
inference:
use_cameras:
– camera_cross_left_120fov
– camera_front_wide_120fov
– camera_cross_right_120fov
– camera_front_tele_30fov
max_batch_size: 1 # A2Super is reminiscence intensive, begin with batch dimension 1
subsample_factor: 1
context_length: 4 # A2Super makes use of 4 temporal frames per digital camera
# Route configuration — A2Super makes use of language-only navigation, not waypoint instructions
route:
use_waypoint_commands: false
# Output configuration
output_dir: “/mnt/output/driver”
# Trajectory optimization (disabled by default for A2Super)
trajectory_optimizer:
enabled: false
plot_debug_images: false
On 913 reconstructed scenes, Alpamayo 2 Tremendous obtains an AlpaSim Rating of 1.50 ± 0.13. This closed-loop rating enhances open-loop outcomes by revealing collisions, highway departures, shut encounters, and different failures that may emerge solely after the coverage influences future observations.
A trajectory is exact, nevertheless it doesn’t all the time present a compact description of intent. Meta-actions summarize a plan by way of high-level choices corresponding to yield, change lanes, cease, or speed up. These outputs may help bridge an end-to-end basis mannequin and a modular AV stack: a downstream planner can eat the choice, an evaluator can verify whether or not trajectory geometry agrees with intent, and groups can search their knowledge corpus for specific maneuvers.
Generate meta-actions
The meta-actions pocket book exhibits how one can produce meta-action outputs with an instance scene. For an in depth record of supported meta-actions, refer to those lists. The core inference step and outputs are proven beneath.
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs
knowledge = load_physical_aiavdataset(
“030c760c-ae38-49aa-9ad8-f5650a545d26”,
t0_us=2000000,
)
mannequin = Alpamayo2Super.from_pretrained(“nvidia/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″)
task_inputs = prepare_text_generation_inputs(
knowledge=knowledge,
model_config=mannequin.config,
tokenizer=mannequin.tokenizer,
job=”meta_action”,
)
task_inputs = helper.to_device(task_inputs, “cuda”)
torch.cuda.manual_seed_all(42)
with torch.autocast(“cuda”, dtype=torch.bfloat16):
outcome = generate_text(
mannequin,
task_inputs,
top_p=0.98,
temperature=0.6,
max_new_tokens=512,
)
cot = outcome[“cot”][0]
meta_action = outcome[“meta_action”][0]
print(“Chain-of-Causation:n”, cot)
print(“nMeta-action:n”, meta_action)
Consider meta-action accuracy
To judge meta-action accuracy, we examine Alpamayo 2 Tremendous’s output with ground-truth labels and report the ensuing classification accuracy by way of intersection-over-union (IoU) for the three elements of its meta-action taxonomy: lateral, longitudinal, and lane-wise.
On an inside set of 94K clips with ground-truth meta-action knowledge, Alpamayo 2 Tremendous achieves 74.59 lateral IoU, 61.91 longitudinal IoU, and 73.55 lane-wise IoU throughout its meta-action taxonomy.
Scene understanding
Planning is just one method to make use of a driving basis mannequin. Visible query answering (VQA) exposes the mannequin’s scene understanding instantly by pure language. Builders can ask about key components within the scene, how they have an effect on driving conduct, why the ego automobile ought to gradual, or about different elements of situations.ne-change hole.
VQA is efficacious for interactive debugging and knowledge operations. It could actually assist a developer examine why a coverage behaved a sure method, construct semantic filters over giant clip collections, and generate candidate annotations for human evaluation. With surround-view enter, questions can reference aspect and rear context {that a} front-view-only mannequin would miss.
Querying multi-camera scenes with VQA and 2D grounding
For every clip, Alpamayo 2 Tremendous can generate solutions and spatially localize referenced actors by predicting 2D bounding bins within the related digital camera frames. This grounding makes outputs extra helpful than free-form textual content alone. Reviewers can confirm which particular object the mannequin is referring to, automated checks can flag lacking or inconsistent bins, and downstream fashions can leverage (e.g., by distillation) these tighter hyperlinks between visible proof, reasoning, and motion.
The scene understanding and VQA pocket book exhibits how one can carry out VQA with the identical inputs. The core inference step and outputs are proven beneath.
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs
knowledge = load_physical_aiavdataset(
“ea4a6729-bf33-4997-905b-cd58774a3580”,
t0_us=7500000,
)
mannequin = Alpamayo2Super.from_pretrained(“nvidia/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″)
task_inputs = prepare_vqa_inputs(
knowledge=knowledge,
model_config=mannequin.config,
tokenizer=mannequin.tokenizer,
query=”Describe the driving scene and establish the important thing site visitors components that ought to affect ego conduct.”,
)
task_inputs = helper.to_device(task_inputs, “cuda”)
with torch.autocast(“cuda”, dtype=torch.bfloat16):
outcome = generate_text(
mannequin,
task_inputs,
top_p=1.0,
temperature=0.1,
max_new_tokens=1024,
)
reply = outcome[“answer”][0]
print(reply)
The mannequin can return two sorts of output for a similar enter: Determine 4, above, exhibits a natural-language description of the scene, whereas Determine 5, beneath, exhibits the identical mannequin spatially localizing referenced objects by predicting 2D bounding bins within the related digital camera body.
Consider VQA response and grounding high quality
To judge VQA efficiency, we examine Alpamayo 2 Tremendous’s generated responses with floor fact solutions throughout an inside set of 8K question-answer pairs. The mannequin achieves 0.652 reply similarity (larger is best), in contrast with Qwen3-VL 32B at 0.450. For 2D grounding, we examine the anticipated bounding bins to ground-truth annotations utilizing IoU and procure 0.71 in comparison with 0.17 for Qwen3-VL 32B. Collectively, these measurements consider whether or not the mannequin each solutions questions precisely and associates its responses with the proper visible proof throughout the surround-view cameras.
Auto-labeling
Reasoning fashions want decision-grounded reasoning knowledge, however labeling long-tail driving clips by hand is dear and gradual. Annotators should examine temporal and multi-camera context, establish the causal actors, describe how they have an effect on the ego automobile, and maintain the label per the supposed maneuver. Alpamayo 2 Tremendous can function an offline auto-labeler that proposes this construction at scale. This could compress annotation cycles from months to days.
Generate structured CoC auto-labels
The CoC auto-labeling pocket book accepts clips within the launched schema and writes one structured file per chosen keyframe.
from alpamayo2_super.load_physical_aiavdataset import load_physical_aiavdataset
from alpamayo2_super.fashions.alpamayo2_super import Alpamayo2Super
from alpamayo2_super.text_tasks import generate_text, prepare_text_generation_inputs
knowledge = load_physical_aiavdataset(
“b5f3756c-4f0e-4298-a1ff-cc92ed392ae0”,
t0_us=11000000,
)
mannequin = Alpamayo2Super.from_pretrained(“nvidia/Alpamayo2-Tremendous”, dtype=torch.bfloat16, device_map=”cuda:0″)
task_inputs = prepare_text_generation_inputs(
knowledge=knowledge,
model_config=mannequin.config,
tokenizer=mannequin.tokenizer,
job=”auto_labeling”,
)
task_inputs = helper.to_device(task_inputs, “cuda”)
torch.cuda.manual_seed_all(42)
with torch.autocast(“cuda”, dtype=torch.bfloat16):
outcome = generate_text(
mannequin,
task_inputs,
top_p=0.98,
temperature=0.6,
max_new_tokens=1024,
)
auto_labeling_text = outcome[“cot_auto_labeling”][0]
auto_labeling_json = outcome[“cot_auto_labeling_json”][0]
print(json.dumps(auto_labeling_json, indent=2))
By default, Alpamayo 2 Tremendous assumes entry to future ego-trajectory info. Nonetheless, it may additionally auto-label knowledge that doesn’t include future trajectory info:
# Get the mannequin to foretell a future trajectory of its personal, and use that in auto-labeling because the “noticed” future movement.
trajectory_inputs = helper.prepare_model_inputs(knowledge, mannequin.config, mannequin.tokenizer)
trajectory_inputs = helper.to_device(trajectory_inputs, “cuda”)
torch.cuda.manual_seed_all(42)
with torch.autocast(“cuda”, dtype=torch.bfloat16):
pred_xyz, pred_rot, _, additional = mannequin.sample_trajectories_from_data(
knowledge=trajectory_inputs,
top_p=0.98,
temperature=0.6,
num_traj_samples=1,
diffusion_kwargs={“inference_step”: 10},
return_extra=True,
)
future_xyz = pred_xyz[:, 0, 0].detach().cpu()
future_rot = pred_rot[:, 0, 0].detach().cpu()
# In case you wish to see the mannequin’s reasoning, uncomment these:
# trajectory_cot = str(additional[“cot”].reshape(-1)[0])
# print(“trajectory_cot:n”, trajectory_cot)
task_inputs = prepare_text_generation_inputs(
knowledge=knowledge,
model_config=mannequin.config,
tokenizer=mannequin.tokenizer,
job=”auto_labeling”,
# That is the place the anticipated trajectories are handed in:
future_xyz=future_xyz,
future_rot=future_rot,
)
task_inputs = helper.to_device(task_inputs, “cuda”)
torch.cuda.manual_seed_all(42)
with torch.autocast(“cuda”, dtype=torch.bfloat16):
outcome = generate_text(
mannequin,
task_inputs,
top_p=0.98,
temperature=0.6,
max_new_tokens=1024,
)
auto_labeling_text = outcome[“cot_auto_labeling”][0]
auto_labeling_json = outcome[“cot_auto_labeling_json”][0]
print(json.dumps(auto_labeling_json, indent=2))
Consider CoC auto-labeling high quality
To judge CoC auto-labeling high quality, we examine Alpamayo 2 Tremendous’s generated labels with knowledgeable annotations on 8k inside clips. We use an inside choose mannequin to evaluate similarity. Alpamayo 2 Tremendous achieves a 0.652 similarity rating, in contrast with Qwen3-VL 32B at 0.450. These outcomes present its potential to generate structured reasoning labels at scale whereas sustaining consistency with expert-authored annotations.
Construct with Alpamayo 2 Tremendous
Alpamayo 2 Tremendous brings surround-view notion, reasoning, planning, scene understanding, and knowledge auto-labeling into one open mannequin workflow. These capabilities give builders a sensible basis for constructing instructor fashions, curating long-tail knowledge, inspecting coverage choices, and evaluating AV methods past a single open-loop trajectory metric. As a instructor mannequin, it may be distilled into compact fashions that run on NVIDIA DRIVE AGX Thor contained in the automobile.
Discover the mannequin on Hugging Face, run the inference notebooks, and share what you construct on the Alpamayo developer discussion board.
For extra particulars on the broader updates launched as a part of the Alpamayo 2 launch, see the related Hugging Face weblog submit.
