Accelerating Transformers Nice-Tuning with NVIDIA NeMo AutoModel

HuggingFace Transformers has turn into the inspiration of the open-source AI ecosystem, and the current Transformers v5 launch strengthened it with first-class help for Combination-of-Consultants (MoE) fashions, now the dominant structure for frontier fashions. v5 ships the MoE foundations: professional backends, dynamic weight loading, and distributed execution that make MoE extensible and simple to construct on.

NVIDIA NeMo AutoModel is an open library a part of the NVIDIA NeMo framework for constructing customized generative AI fashions at scale. NeMo AutoModel builds cleanly on prime of v5, including Professional Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels, and it leans on v5’s dynamic weight loading to deliver these optimizations to a broad and rising set of mannequin households. The payoff is 3.4-3.7x larger coaching throughput and 29-32% much less GPU reminiscence on fine-tuning MoE fashions than native Transformers v5, utilizing the identical from_pretrained() API: a single import line, with no different code adjustments.

This weblog particulars how this mix works and the way customers can fine-tune MoE fashions sooner with out altering their APIs.

The rise of MoE fashions has launched new challenges to environment friendly coaching: Routing tokens throughout a whole lot of consultants, fusing professional matmuls right into a single kernel, sharding weights throughout GPUs, and overlapping communication with computation all require infrastructure past what a general-purpose library gives out of the field.

Transformers v5 (“v5”) launched first-class MoE help corresponding to professional backends, dynamic weight loading, and tensor parallel plans for distributed execution. As well as, v5 made distributed coaching first-class by integrating PyTorch’s DeviceMesh instantly into from_pretrained().

NeMo AutoModel builds on prime of v5 by subclassing AutoModelForCausalLM, and including Professional Parallelism (EP), DeepEP fused all-to-all dispatch, and TransformerEngine kernels. DeepEP is the piece v5 does not have but: it overlaps communication with professional compute. And since NeMo AutoModel rides v5’s reversible weight conversion to load every mannequin, it will probably focus its engineering on these reusable core ops as an alternative of per-model checkpoint plumbing, whereas save_pretrained() nonetheless emits commonplace HF checkpoints that instruments like vLLM and SGLang can load.

The following part walks by means of how the 2 work collectively and the efficiency good points we measured, from full fine-tuning NVIDIA Nemotron 3 Extremely 550B A55B throughout 16 nodes all the way down to single-node fashions corresponding to Qwen3-30B-A3B and Nemotron 3 Nano 30B A3B.

NeMo AutoModel: Similar API, Extra Efficiency

Considered one of NeMo AutoModel’s targets is API compatibility with HuggingFace Transformers to allow open-source neighborhood. NeMoAutoModelForCausalLM subclasses AutoModelForCausalLM, so any code that works with HF fashions works with AutoModel too.

Here is what loading a mannequin seems to be like in each. Solely the import adjustments:

That single import does lots of work. For well-liked MoE architectures like Qwen3, NVIDIA Nemotron, GPT-OSS, and DeepSeek V3, NeMo AutoModel ships hand-tuned implementations with TransformerEngine consideration, fused linear layers, and customized professional kernels. For the whole lot else, it falls again to vanilla HF whereas nonetheless making use of optimizations like Liger kernel patching, amongst others. And whichever path it takes, the ensuing mannequin is able to scale: move a device_mesh and you’ve got multi-GPU coaching with out additional rewrites.

The place NeMo AutoModel actually shines is scaling MoE fashions to multi-GPU coaching. To coach Nemotron 3 Nano 30B A3B with Professional Parallelism throughout 8 GPUs, one provides the distributed mesh configuration:

import os
import torch
import torch.distributed as dist
from nemo_automodel import NeMoAutoModelForCausalLM
from nemo_automodel.recipes._dist_utils import create_distributed_setup_from_config

dist.init_process_group(backend=“nccl”)
torch.manual_seed(0)
torch.cuda.set_device(int(os.environ.get(“LOCAL_RANK”, 0)))

dist_setup = create_distributed_setup_from_config(
{
“technique”: “fsdp2”,
“ep_size”: 8,
},
)

mannequin = NeMoAutoModelForCausalLM.from_pretrained(
“nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16”,
dtype=torch.bfloat16,
distributed_setup=dist_setup,
)

dist.destroy_process_group()

This offers velocity, scalability and memory-optimizations with FSDP2, Professional Parallelism, TransformerEngine kernels and DeepEP dispatch, all from a from_pretrained() name.

Efficiency Comparability

We evaluated NeMo AutoModel in two regimes: full fine-tuning a frontier-scale 550B mannequin throughout 16 nodes, and coaching two 30B MoE fashions on a single node. The 550B outcome reveals why Professional Parallelism is important at scale; the 30B outcomes quantify the per-GPU speedup over Transformers v5.

Nemotron 3 Extremely 550B A55B (full fine-tune, multi-node)

Nemotron 3 Extremely 550B A55B is a 550B-parameter hybrid mannequin transport with Mamba2, LatentMoE, and Multi-Token Prediction (MTP). We benchmark a full fine-tune: each parameter is up to date and the Adam optimizer state is materialized, which at this scale spans 16 H100 nodes (128 GPUs).

Methodology:

Why there is no such thing as a Transformers v5 column. Transformers v5 runs out of reminiscence at this scale, so there is no such thing as a v5 quantity to report right here. AutoModel’s Professional Parallelism shards the consultants throughout GPUs to deliver the footprint inside finances, which is what lets the complete fine-tune run. The 30B comparisons under present the identical benefit the place v5 matches.

Single-node 30B MoE benchmarks

We benchmarked three approaches on a single node with 8x H100 80GB GPUs: HF Transformers v4 (hub code), HF Transformers v5 (with greatest obtainable optimizations), and NeMo AutoModel (EP=8 + customized kernels).

Methodology:

A notice on the routing gate. The NeMo AutoModel numbers under use a balanced routing gate, which forces tokens to be distributed uniformly throughout consultants. This emulates the perfect working level an MoE is skilled towards: a well-trained mannequin’s load-balancing loss drives professional utilization to near-uniform, so balanced routing displays the steady-state an actual workload converges to (and removes the straggler noise that random dummy tokens in any other case inject into professional parallelism). v4/v5 run their native router on the identical dummy tokens. The balanced gate subsequently measures NeMo AutoModel at its goal MoE working level, and the v4/v5 columns mirror their out-of-the-box conduct.

Qwen3-30B-A3B

Why v4 deadlocks: Transformers v4 shops Qwen3 MoE consultants as a ModuleList of 128 particular person MLP modules, every individually FSDP-wrapped. The ahead move makes use of a data-dependent loop that solely iterates consultants that acquired tokens. With completely different knowledge per rank, completely different ranks skip completely different consultants, inflicting mismatched FSDP AllGather/ReduceScatter collectives and an indefinite hold. Transformers v5 fixes this by storing consultants as fused 3D parameter tensors (no per-expert modules, no per-expert FSDP collectives).

Nemotron 3 Nano 30B A3B

v4 config: trust_remote_code=True (NVIDIA’s hub modeling code). The hub code’s professional loop is FSDP-safe (iterates all consultants no matter token project), so it does not impasse like Qwen3 v4.

The place the speedup comes from

The three.4-3.7x speedup from NeMo AutoModel over Transformers v5 comes from three sources:

Professional Parallelism reduces reminiscence strain. EP=8 distributes professional weights throughout GPUs, reducing the per-GPU MoE footprint by 8x. For Qwen3, this drops peak reminiscence from 68.2 GiB to 48.1 GiB (-29%). For Nemotron Nano, it drops from 62.1 GiB to 42.5 GiB (-32%), liberating headroom for bigger batch sizes or longer sequences.

DeepEP fuses communication with computation. As a substitute of separate AllGather/ReduceScatter collectives for professional routing, DeepEP fuses token dispatch and combines into optimized GPU kernels, overlapping communication with professional computation.

TransformerEngine kernels speed up core operations. TE’s fused consideration, linear layers, and RMSNorm implementations present constant speedups over their PyTorch/Flash Consideration equivalents throughout all layer varieties, not simply MoE layers.

Transformers v5 Options Leveraged by HuggingFace AutoModel

Professional Backends

Some of the impactful options in Transformers v5 is the experts_implementation parameter, which incorporates three professional backends:

The grouped_mm backend is the important thing coaching optimization: as an alternative of looping over consultants one after the other, it kinds tokens by their assigned professional and executes a single fused grouped matrix multiplication.

NeMo AutoModel takes this additional. For fashions with customized implementations, it makes use of DeepEP fused all-to-all dispatch mixed with grouped GEMM kernels and TransformerEngine linear layers. The development seems to be like:

v4 (keen for-loop) → v5 (grouped_mm) → NeMo AutoModel (DeepEP + GMM + TE)

In NeMo AutoModel, the professional backend is configured by means of BackendConfig:

from nemo_automodel.elements.fashions.frequent.utils import BackendConfig

backend = BackendConfig(
attn=“te”,
linear=“te”,
consultants=“torch_mm”,
dispatcher=“deepep”,
)

Professional Parallelism and DeepEP

Transformers v5 additionally ships an Professional Parallelism path. It shards professional weights throughout GPUs. The GroupedGemmParallel model masses solely every system’s native consultants, and RouterParallel routes tokens and combines outcomes with an all_reduce. It is neatly constructed on v5’s present tensor-parallel equipment. Enabling it makes the mannequin’s tp_plan return its professional plan, so professional parallelism shares the system finances with knowledge parallelism (ep × dp = world_size). For the single-node 30B benchmarks right here, we discovered plain data-parallel v5 (dp=8, ep=1) to be the quickest v5 configuration, so that is the v5 setup we report.

NeMo AutoModel takes a complementary method tuned for multi-GPU MoE coaching. It makes EP its personal parallelism dimension, a devoted moe_mesh alongside (slightly than carved from) the data-parallel mesh, utilizing PyTorch’s DTensor with Shard(0). As a result of the professional mesh is orthogonal to knowledge parallelism, the 2 compose on the identical gadgets. On 8 GPUs NeMo AutoModel runs ep=8 and dp=8 collectively, so each GPU trains by itself knowledge shard whereas holding just one/8 of the consultants. Professional weights are bodily sharded throughout GPUs alongside the professional dimension.

from torch.distributed.tensor import Shard, distribute_tensor

distribute_tensor(param, device_mesh, [Shard(0)])

With ep_size=8 on 8 GPUs, every GPU holds just one/8 of the professional parameters. For a mannequin like Nemotron-3-Nano-30B-A3B with ~55 GiB of professional weights, EP reduces the per-GPU professional footprint from ~55 GiB to ~6.8 GiB, making coaching potential the place FSDP-only approaches run out of reminiscence.

On prime of EP, NeMo AutoModel integrates DeepEP that fuses the token routing into optimized GPU kernels, and delivers vital speedups when mixed with grouped GEMM for grouped professional computation. In our large-scale MoE benchmarks, DeepEP + grouped GEMM diminished price per iteration by 47% on the complete DeepSeek V3 671B mannequin in comparison with all-gather + looped professional baselines.

Dynamic Weight Loading

Transformers v5 additionally launched a dynamic weight loading system by means of WeightConverter and WeightRenaming. This permits MoE checkpoint to be saved in fused 3D tensors for extra environment friendly execution. The WeightConverter applies composable operations to remodel checkpoint tensors on-the-fly throughout from_pretrained().

NeMo AutoModel is a direct client of this v5 API. Over 20 mannequin varieties use this mechanism by means of MODELS_REQUIRING_TENSOR_MERGING, together with Mixtral, Qwen2 MoE, Qwen3 MoE, DeepSeek V2/V3, OLMoE, and extra. The conversions are totally reversible: save_pretrained() produces commonplace HF-format checkpoints that any downstream instrument can load.

Getting Began

To strive NeMo AutoModel, please go to our official documentation web page to get began.

For extra particulars, see:

Conclusion

NVIDIA NeMo AutoModel is the pure subsequent step for HuggingFace customers scaling up mannequin coaching. By constructing instantly on Transformers v5, AutoModel gives a zero-friction improve path: change one import line and get a mannequin occasion that’s greater than 3 times as quick.

On Qwen3-30B-A3B and Nemotron 3 Nano 30B-A3B, this delivers 3.4-3.7x larger coaching throughput with 29-32% much less GPU reminiscence in comparison with the very best Transformers v5 configuration. And since true Professional Parallelism shards consultants throughout GPUs, the identical path scales as much as full fine-tuning a 550B mannequin like Nemotron 3 Extremely throughout 16 nodes, the regime the place Professional Parallelism turns into important to suit the mannequin in reminiscence. As a result of NeMo AutoModel checkpoints are commonplace HF-format safetensors, you’ll be able to deploy them on inference frameworks like vLLM and SGLang.

The code, configs, and benchmark scripts are all obtainable within the NeMo AutoModel repository.

Acknowledgements

Core contributors to this work, listed alphabetically by final title: Adil Asif, Hemil Desai, Alexandros Koumparoulis, and Huiying Li.