Methods to Submit-Prepare Autonomous Automobile Fashions in Closed-Loop with NVIDIA Alpamayo

Growing autonomous automobile (AV) insurance policies requires bridging an necessary hole between coaching and deployment. Imaginative and prescient-language-action (VLA) fashions that may purpose over extra advanced driving scenes and produce richer intermediate reasoning are predominantly educated in open-loop, the place mannequin outputs are instantly in comparison with ground-truth behaviors with out contemplating their impact on the surroundings.

In deployment, nonetheless, a driving coverage runs in closed-loop, the place each braking, steering, and navigation resolution impacts the surroundings, and small errors can compound over time.

A scientific means to handle this problem is supplied by NVIDIA Alpamayo, an open portfolio of AI fashions, simulation frameworks, and bodily AI datasets for AV growth. Alpamayo consists of the AlpaSim AV simulation platform and the AlpaGym closed-loop coaching framework (coming quickly).

This put up explains easy methods to prepare AV fashions in closed-loop with NVIDIA Alpamayo. Particularly, it walks by way of easy methods to:

Set up and configure AlpaGym 

Outline closed-loop rewards

Launch closed-loop coaching

Export the post-trained checkpoint for downstream use

Closed-loop post-training with AlpaGym extends AV coaching workflows by turning AlpaSim rollouts into coaching expertise. Reasonably than treating simulation solely as a last analysis stage, AlpaGym connects simulator suggestions on to the coverage coaching loop.

Methods to use AlpaGym for closed-loop reinforcement studying

Reinforcement studying (RL) can be utilized to enhance a coverage that was initially educated in open-loop. As an alternative of optimizing solely in opposition to logged knowledgeable trajectories, the mannequin can now study from the results of its personal actions in simulation.

This shift is important for AV growth, the place small prediction or planning errors can compound over time. In closed-loop coaching, every braking, steering, and navigation resolution impacts the following state of the surroundings, revealing failure modes that static datasets or open-loop analysis might miss.

Nonetheless, enabling closed-loop RL comes with its personal challenges. Mannequin inference, working simulation, coaching fashions, syncing weight updates, speaking throughout situations and shifting knowledge—all in parallel—is advanced. This requires orchestration and environment friendly utilization of compute assets in a sturdy but versatile method. 

To deal with these challenges, AlpaGym connects coverage coaching to AlpaSim closed-loop rollouts and gives an open supply, high-throughput framework for closed-loop RL. The system combines AlpaSim simulator microservices, NVIDIA Bodily AI Open Datasets, and distributed NVIDIA Cosmos-RL coaching framework right into a scalable post-training pipeline.

Constructed to scale seamlessly from a single GPU to multi-node GPU clusters, AlpaGym helps environment friendly large-scale coaching by way of an asynchronous and steady distributed RL pipeline, with out requiring adjustments to person code. It integrates AlpaSim and Cosmos RL as its runtime and orchestration layer, GRPO as a default algorithm, and consists of reference reward features examined with Alpamayo fashions and the Bodily AI AV NuRec dataset.

To get began with AlpaGym post-training, observe the steps outlined beneath.

Step 1: Set up and configure AlpaGym

To put in AlpaGym from the Alpamayo checkout, set up the native CUDA dependencies and Redis on the host, then sync the UV workspace:

The Python surroundings is managed by uv, however cuDNN, NCCL, and the redis-server binary are host dependencies utilized by the CUDA mannequin stack and Cosmos-RL. Alternatively, an appropriate Dockerfile can be supplied. Hugging Face authentication is required to obtain the scene artifacts.

An AlpaGym run is a Hydra configuration. It specifies the coverage checkpoint, the AlpaSim scene set, rollout parallelism, reward perform, and Cosmos-RL coaching parameters. On this workflow, the beginning checkpoint is an Alpamayo mannequin.

Step 2: Outline the closed-loop reward

The reward ought to match the habits you need to enhance in closed-loop. For trajectory-quality post-training, widespread reward phrases embrace progress, lane preserving, collision avoidance, offroad fee, consolation, and distance to a reference trajectory.

A sensible first reward is deliberately easy: mix progress with penalties for safety-critical failures. In AlpaGym, this may be expressed as a small sum of phrases, utilizing AlpaSim metrics the place doable:

As soon as the pipeline is steady, add extra focused phrases for the failure modes noticed in AlpaSim movies and metrics.

Step 3: Launch closed-loop post-training

Begin AlpaGym coaching out of your mannequin checkpoint. Alpamayo serves for example mannequin right here.

It will carry up AlpaGym with AlpaSim on a single GPU. Keep tuned for detailed directions on easy methods to use your personal AV mannequin.

Throughout coaching, AlpaGym requests scene rollouts from AlpaSim, collects per-episode artifacts, computes rewards, and updates the coverage. Helpful coaching alerts embrace imply reward, reward variance, failure charges, coverage loss, rollout throughput, and the hole between generated rollouts and the newest coverage weights.

On this recipe, these rollout artifacts and coaching alerts are the first outputs of the post-training run. They assist you verify that closed-loop studying is working appropriately and choose checkpoints for downstream analysis by yourself held-out AlpaSim state of affairs suites.

Step 4: Export the post-trained checkpoint

After coaching, place the AlpaGym-produced checkpoint and config information right into a folder that may be accessed by the AlpaSim driver (your Hugging Face mannequin cache, for instance). Then create a brand new driver config with that folder path (known as alpamayo1_CLRL right here). See the next code for what to edit to specify customized paths in a driver yaml config. This makes the AlpaGym post-trained coverage runnable inside AlpaSim for closed-loop rollouts.

Subsequent, run the exported mannequin on a consultant state of affairs to confirm that the coverage, driver, and simulation loop are linked appropriately. At this stage, you may examine how the coverage behaves when its personal actions have an effect on the following state of the surroundings.

A closed-loop rollout gives helpful qualitative alerts: whether or not the mannequin produces steady trajectories and stays throughout the drivable space, the way it reacts to close by site visitors brokers, and which failure modes must be focused throughout post-training.

With this checkpoint, groups can examine rollout movies, per-episode metrics, reward traces, and failure circumstances collected throughout coaching. These artifacts are helpful for debugging reward design, checking rollout stability, and deciding on checkpoints for later held-out analysis in AlpaSim.

Get began post-training AV fashions 

Closed-loop post-training gives a sensible path for iterating on end-to-end driving insurance policies. On this case, AlpaGym makes use of closed-loop rollouts to post-train AV insurance policies in simulation, enabling them to study from the results of their actions.

You need to use these instruments along with the opposite elements of the NVIDIA Alpamayo Open Platform to develop reasoning fashions that may be run, inspected, and post-trained in a closed-loop simulation workflow. Lengthen this similar recipe extra broadly with your personal rewards, situations, and analysis suites.

Able to get began? Try the NVlabs/alpamayo-recipes GitHub repo to adapt the recipe on this put up on your personal use circumstances. 

To guage your mannequin on a public leaderboard, see the 2 open AV challenges NVIDIA launched at CVPR 2026: 

To study extra, see Increasing the Alpamayo Open Platform for Growing Reasoning AVs Throughout Fashions, Knowledge, and Simulation.

Be part of NVIDIA founder and CEO Jensen Huang for the NVIDIA GTC Taipei 2026 Keynote and dive deeper with associated classes.