Groups customise their fashions to hit their targets for latency, pace, reminiscence, and compute. With the open NVIDIA Nemotron household of fashions, builders can discover the right-sized mannequin for his or her wants.<\/p>\n
The brand new Nemotron 3.5 Lightning NVFP4 checkpoint, for instance, preserves accuracy whereas unlocking as much as 4x sooner throughput. It\u2019s compressed right down to 22 GB from the 66 GB full precision checkpoint by quantizing lots of its weights to 4 bits.<\/p>\n
To compress fashions to NVFP4, post-training quantization (PTQ) is a typical technique that covers most wants. However if you wish to attain excessive throughput with tighter reminiscence, extra aggressive quantization is required. Quantization-aware distillation (QAD) is an optimum alternative on this case. Coaching Nemotron 3.5 Lightning to adapt to quantization noise with QAD produced an NVFP4 checkpoint that makes use of much less reminiscence and delivers increased throughput whereas preserving accuracy.<\/p>\n
This submit demonstrates how QAD improves the Nemotron 3.5 Lightning mannequin utilizing NVIDIA Mannequin Optimizer. We stroll by your complete coaching pipeline, from the preliminary PTQ section to the ultimate distillation and analysis. We present that QAD recovers accuracy degradation from aggressive quantization. Even with extra conservative configurations, QAD constantly outperforms PTQ on agentic benchmarks, guaranteeing prime quality whereas lowering reminiscence utilization. <\/p>\n
What’s quantization-aware distillation?<\/h2>\n
QAD makes use of the unique full-precision mannequin (trainer) to show the quantized mannequin (pupil). First, create a quantized mannequin by working PTQ on the full-precision mannequin. Then distill the frozen BF16 mannequin into the quantized mannequin utilizing a KL divergence loss evaluating the trainer\u2019s and pupil\u2019s logits.<\/p>\n
Determine 1 exhibits the two-stage QAD course of used to construct the Nemotron 3.5 Lightning NVFP4 checkpoint. The complete-precision BF16 mannequin serves because the frozen trainer and can also be the start line for Stage 1, a PTQ go that quantizes weights to W4A16 to provide the quantized pupil. In Stage 2, the scholar is educated with QAD, working its ahead go by simulated quantization whereas a distillation loss aligns it with the trainer, yielding the ultimate NVFP4 checkpoint with accuracy recovered near baseline.<\/p>\n
![\"A](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/08\/nvfp4-qad-nemotron-3.5-lightning.webp\")
The quantization-aware distillation course of<\/h2>\n
Comply with the steps beneath to execute the QAD course of.<\/p>\n
Step 1: Submit-training quantization<\/h3>\n
The primary stage of QAD is working PTQ to provide a quantized checkpoint (pupil). Since we plan to run QAD, extra aggressive quantization will be carried out. For Nemotron 3.5 Lightning, we discovered that quantizing the Mamba linear layers to a extra aggressive W4A16, quite than FP8, unlocked increased throughput with out a big drop in accuracy.<\/p>\n
Usually, a median accuracy restoration of over 99% is focused when performing PTQ solely. When mixed with QAD a goal 95-99% median accuracy restoration will be focused as a result of QAD will get better extra accuracy. This confirms that the quantization has been pushed far sufficient to financial institution the dimensions and latency good points, whereas leaving clear room for QAD to shut the hole within the subsequent stage.<\/p>\n
We count on a small however significant drop on the analysis benchmarks with W4A16 however plan to make use of QAD to get better the drop. This confirms that the quantization has been pushed far sufficient to financial institution the dimensions and latency good points, whereas leaving clear room for QAD to shut the hole within the subsequent stage.<\/p>\n
Step 2: Quantization-aware distillation<\/h3>\n
Throughout QAD, each ahead go of the scholar mannequin runs by simulated quantization so the mannequin can account for the quantization noise it can encounter at inference. On the identical time, it’s educated to match the trainer by a distillation loss. By means of the trainer sign, the scholar is studying to breed the complete habits of the mannequin it got here from quite than simply predicting the following token. Coaching towards each alerts collectively permits QAD to take care of prime quality with aggressive quantization.<\/p>\n
To be taught extra concerning the QAD course of, see the end-to-end examples of QAD on NVIDIA Mannequin Optimizer.<\/p>\n
How one can develop Nemotron 3.5 Lightning NVFP4 with QAD utilizing Mannequin Optimizer <\/h2>\n
The next sections clarify the method we used to develop the Nemotron 3.5 Lightning NVFP4 checkpoint utilizing QAD with NVIDIA Mannequin Optimizer.<\/p>\n
Step 1: Receive a PTQ checkpoint<\/h3>\n
The bottom mannequin, NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, is the trainer. To create the scholar, run PTQ on the identical base to quantize it to W4A16-NVFP4. Since each come from the identical mannequin, the scholar has a well-matched trainer to be taught from.<\/p>\n
We tried a number of PTQ recipes for the scholar, which differ in how the weights are calibrated and the way aggressively the Mamba projections and KV cache are quantized. This alternative carries into coaching: max-calibrated recipes feed dynamic scale QAD, whereas MSE-based recipes feed frozen-scale QAD (see Step 2 of this part).<\/p>\n
A number of settings are shared throughout each recipe. All of them quantize the lm_head to W4A16, a alternative we name a trustworthy lm_head, whereas consideration projection layers keep in BF16. Calibration makes use of 1,000 samples and runs on a single NVIDIA DGX B300. The final recipe, four_over_six plus NVFP4 KV, is probably the most aggressive. It pushes solely Ok and V to NVFP4 (W4A4) and leaves the (QK^{mathsf{T}}) and (mathrm{attn} cdot V) batched matrix multiplications in BF16, with Q left unquantized.<\/p>\n
It\u2019s advantageous to quantize extra aggressively on the PTQ stage than if PTQ had been the ultimate step, as a result of QAD recovers accuracy afterward. This allows you to choose settings a PTQ-only recipe would keep away from, akin to taking the Mamba linear layers to W4A16 as an alternative of the safer FP8.<\/p>\n
In reality, we wish the PTQ checkpoint to indicate a small however significant drop on the analysis benchmarks, bringing median accuracy restoration between 95 to 99%. This drop confirms we pushed exhausting sufficient to financial institution the dimensions and latency good points, whereas leaving clear room for QAD to shut the hole within the subsequent stage.\u00a0<\/p>\n
We evaluated many PTQ recipes, specializing in the 5 recipes beneath. For every recipe, we carried out PTQ calibration with sequence size starting from 8k to 128k. Prior experiments recommend longer sequence size produces higher PTQ outcomes. We discovered {that a} 32K sequence size offered the most effective PTQ outcomes on four_over_six and carried out all QAD experiments on the 32K calibration four_over_six checkpoint. After evaluating completely different PTQ recipes, we discovered the four_over_six with W4A16 Mamba linears offered the most effective tradeoff of accuracy degradation to spice up in inference efficiency. For full accuracy particulars, see the QAD Checkpoint evaluations part.<\/p>\n
Desk 1 exhibits the 5 PTQ recipes used to construct the Nemotron 3.5 Lightning pupil checkpoints. Every recipe is outlined by its weight format for the MoE, shared, and lm_head layers, its calibration technique, its Mamba in\/out projection format, and its KV cache format. All 5 use W4A16 NVFP4 weights and vary from max-calibrated dynamic recipes to MSE-based static recipes, with probably the most aggressive variant pushing the KV cache to NVFP4. Mild Inexperienced rows use dynamic scaling and lightweight grey rows use static scaling within the coaching step.<\/p>\n You’ll be able to reproduce this recipe by yourself mannequin utilizing NVIDIA Mannequin Optimizer. The Hugging Face PTQ instance walks by quantizing a Hugging Face mannequin to NVFP4 utilizing the completely different PTQ recipes beforehand defined.<\/p>\n # outline ahead loop with dataset # Quantize base mannequin to NVFP4 to create the PTQ pupil checkpoint The identical Hugging Face PTQ instance additionally covers the calibration information, supported codecs, and export choices. After getting a PTQ checkpoint, you possibly can transfer on to the QAD coaching configuration and scale technique beforehand described.<\/p>\n This part explains the QAD coaching, together with configuration and quantization scale dealing with.<\/p>\n With the scholar checkpoint in hand, the following step is deciding methods to practice it. Two decisions mattered most in our ablations: the sequence size for coaching and the info for distillation.<\/p>\n Sequence size: Sequence size turned out to be essential for sure benchmarks, particularly the longer-context ones, the place coaching too brief leaves accuracy on the desk. Submit-training supervised fine-tuning (SFT) used roughly 522K tokens, and our ablations confirmed that 522K sequence size was needed for preserving long-context efficiency. To stability compute sources with sequence size, we carried out preliminary ablations with 256K sequence size, after which scaled as much as 522K for the ultimate QAD run.<\/p>\n Datasets: For the info combine, we ablated a couple of inner mixes earlier than touchdown on our ultimate recipe. For anybody trying to reproduce this work, we suggest ranging from the open datasets NVIDIA has launched, Nemotron-Submit-Coaching v1 and Nemotron-Submit-Coaching v2, which cowl an analogous distribution to what we used.<\/p>\n Distillation recipe: The coed begins from the PTQ checkpoint: the BF16 mannequin is quantized first to NVFP4 utilizing the Nemotron-3.5-Lightning-30B-A3B\/lightning quantization recipe. Then distill from there quite than coaching the quantized weights from scratch. On every step, the identical batch runs by each the BF16 trainer and the NVFP4 pupil, and the scholar is educated to match the trainer by a KL divergence distillation loss on the logits, with the trainer stored frozen because the reference sign. Distill at a relentless studying charge of 5e-6 with no warmup, dropout disabled, and gradient clipping at 1.0, throughout two nodes occasions 8 GPUs (TP=2, EP=4). This is identical QAD workflow that ships with the NVIDIA Mannequin Optimizer.<\/p>\n With the checkpoint and coaching configuration set, the final alternative is methods to deal with the quantization scales throughout QAD. We tried two methods, and the appliance relies on how the PTQ checkpoint from Step 1 was calibrated.<\/p>\n Determine 2 exhibits dynamic versus frozen scale, through which each lanes share the identical quantized ahead go consisting of activations, simulated quantization, GEMM, loss with gradients returning to the FP weights. The highest lane recomputes its quantization scale every step from the present tensors; the underside lane retains the PTQ-calibrated scale locked and updates solely the weights. W(t) is the BF16 weights at coaching step t, s(t) is the dimensions recomputed from these weights at that very same step, and s* is that scale captured as soon as and held for the entire run.<\/p>\n Dynamic scale QAD: Begins from a max calibrated PTQ checkpoint. That is both the max or mamba_fp8_max quantization config from Desk 1. Each carry dynamic weight and activation scales, so the scales are recomputed on the fly throughout coaching and each the weights and the scales adapt because the mannequin learns.<\/p>\n Frozen scale QAD: Begins from an MSE primarily based PTQ checkpoint, akin to MSE, four_over_six, or four_over_six + NVFP4 KV from Desk 1. MSE and 4 over six arrive at their scales by a search to attenuate quantization error, which is way too costly to repeat at each step. As an alternative of recomputing the scales,\u00a0 take the scales discovered throughout PTQ and freeze them throughout coaching. Solely the weights are up to date, whereas the scales keep fastened at their calibrated values.\u00a0<\/p>\n The size technique is chosen earlier than coaching time. It follows straight from the PTQ recipe chosen in Step 1. Max-calibrated checkpoints result in dynamic scale QAD, and MSE-based checkpoints result in frozen scale QAD. It may be a good suggestion to strive a number of PTQ recipes with a mixture of dynamic and frozen scales, and consider them to evaluate which one will profit from QAD.<\/p>\n The next experiments on intermediate Lightning checkpoints show how QAD permits for extra aggressive quantization than PTQ alone. Each the PTQ and QAD checkpoints use the identical aggressive W4A16 quantization format recipe and the identical 21.19 GB footprint, towards 65.85 GB for the BF16 baseline, so any distinction between them comes from the tactic quite than from additional reminiscence. QAD runs on intermediate checkpoints A and B had been carried out on 256K sequence size for speedy experimentation, whereas the ultimate checkpoint QAD used 524K sequence size.<\/p>\n Checkpoint A (Desk 2) is an SFT intermediate checkpoint. Checkpoint B (Desk 3) is an intermediate RL checkpoint. We evaluated these checkpoints and located that QAD can get better some accuracy loss from aggressive quantization on solely PTQ.<\/p>\n We tried QAD first on an early pre-release Checkpoint A, quantized extra aggressively than what ultimately shipped: W4A16 pushed during the Mamba linear projections. Submit-training quantization alone landed at 96.33% median accuracy restoration, with the most important regressions falling on the reasoning and coding benchmarks, the place AIME 2025 dropped 3.70 factors, SciCode Subtask 2.85, and HLE 2.36. 200 iterations of QAD from that very same checkpoint introduced restoration to 99.72%, returning AIME 2025 to inside 0.57 factors of BF16 and inserting each SciCode Subtask and HLE barely above it.<\/p>\n Desk 3 exhibits checkpoint B, scored on the up to date analysis suite, with the identical end result. PTQ reaches 95.84% median rating restoration and QAD reaches 98.53%, once more at an an identical 21.19 GB. The AA v4.1 Index is the clearest single measure right here, rising 3.45 factors from 20.03 to 23.48 towards a BF16 baseline of 24.81. QAD improves 5 of the 9 benchmarks, so the median quite than any particular person rating carries the end result.<\/p>\n We repeated the experiment on a later Checkpoint B and noticed the identical sample: 95.84% to 98.53% median restoration. We additionally measured the AA Index, which rose 3.4 factors from 20.03 to 23.48 towards a BF16 baseline of 24.81. The launch NVFP4 recipe is intentionally extra conservative, so PTQ already sits near baseline and QAD has significantly much less to get better, with the remaining good points narrowing to a subset of the agentic benchmarks. Total, QAD recovers the accuracy loss that aggressive quantization introduces. PTQ fixes the weights to a low-precision grid and accepts no matter error that produces, whereas QAD permits the mannequin to proceed coaching towards that error and adapt to it, so a lot of the degradation is recovered quite than absorbed.<\/p>\n That distinction determines how far a recipe will be pushed. Configurations that PTQ leaves a number of factors beneath baseline turn into viable underneath QAD, which suggests the reminiscence and throughput budgets that may in any other case drive a extra conservative precision at the moment are inside attain.<\/p>\n Desk 4 exhibits accuracy evaluations for the ultimate NVFP4 checkpoint BF16 versus PTQ versus QAD. The ultimate checkpoint was quantized conservatively to optimize accuracy, so PTQ already sits near BF16 and there’s little loss left for QAD to get better. Median rating restoration is 99.24% for PTQ and 98.97% for QAD. The good points present up on the agentic and coding benchmarks as an alternative, the place QAD beats PTQ by 3.79 factors on Terminal-Bench v2.1, 1.07 on SWE-Bench Multilingual, 0.65 on HLE and smaller margins on BrowseComp, SWE-Bench Verified, and PinchBench.<\/p>\n For this ultimate checkpoint, we optimized for accuracy, that means extra conservative quantization utilizing Nemotron-3.5-Lightning-30B-A3B\/lightning_w4a16_nvfp4_4o6. PTQ due to this fact lands near the BF16 baseline and QAD has a lot much less loss to get better, with median rating restoration at 99.24% and 98.97%, respectively.<\/p>\n The good points from QAD are smaller than underneath aggressive PTQ, however they present up the place it issues: QAD recovers accuracy on a number of key agentic benchmarks, together with Terminal-Bench v2.1, SWE-Bench Multilingual, BrowseComp, PinchBench, HLE, and AA-Omniscience accuracy.<\/p>\n The complete QAD recipe ships in NVIDIA Mannequin Optimizer as a single launcher megatron_lm_qad.yaml that runs the entire pipeline end-to-end. The identical file works for each Nemotron 3 Nano and Nemotron 3.5 Lightning. Level it on the mannequin you need and launch. The ultimate NVFP4 checkpoint is printed on Hugging Face.<\/p>\n Step one right here is to make use of PTQ to quantize the BF16 trainer into the NVFP4 pupil. Nemotron 3.5 Lightning doesn’t use RoPE, so –max-position-embeddings has no impact on positional encoding right here. Right here we preserve it equal to its most context size (1M) for consistency. For fashions that do use RoPE, go away it on the config.json worth.<\/p>\n QAD is enabled by setting the trainer by –export-kd-teacher-load with –modelopt-enabled added. Underneath the hood, the launcher invokes the Megatron-LM coaching entry level, finetune.sh, which runs the precise distillation loop towards the frozen trainer.<\/p>\n To breed our setup, level the launcher at your PTQ pupil checkpoint and trainer, regulate the sequence size and information combine to match your goal, and launch. We distill at a relentless 5e-6 studying charge with dropout off and gradient clipping at 1.0, on a single Nemotron-Submit-Coaching v2 chat shard:<\/p>\n A ultimate export activity converts the educated pupil right into a deployable Hugging Face NVFP4 checkpoint.<\/p>\n To breed by yourself mannequin, swap the mannequin paths and select your QUANT_CFG (which units the matching scale technique from Step 3); to coach at the next sequence size, set each \u2013seq-length and \u2013max-position-embeddings to the next worth, akin to 524,288 (524k). For the whole launcher, see megatron_lm_qad.yaml.<\/p>\n The identical steps additionally run by Megatron-Bridge, a PyTorch-native library inside the NeMo Framework that gives pretraining, SFT, and LoRA for well-liked language, vision-language, audio, and multimodal mannequin help. It serves as a bridge between Hugging Face and Megatron Core offering bidirectional checkpoint conversion between the 2 codecs. Initiatives can leverage Megatron Core parallelism capabilities or export fashions for numerous inference engines, with built-in verification.<\/p>\n The mbridge_qad.yaml launcher runs your complete course of end-to-end as 4 duties: tokenize the coaching information, quantize the BF16 mannequin into the NVFP4 pupil, distill that pupil towards the frozen trainer, and export a deployable Hugging Face checkpoint. Right here we use pre-tokenized information, the Nemotron-Submit-Coaching-Dataset-v2 chat cut up is tokenized as soon as up entrance with megatron_preprocess_data and handed to distillation by –data_paths.<\/p>\n This runs on eight nodes with 4 GPUs (TP=1, PP=1, CP=4, EP=16), giving DP=8, with microbatch dimension=1 and world batch dimension=64; 200 iterations cowl roughly 419M coaching tokens.<\/p>\n The pipeline contains the three steps detailed on this submit, wrapped in 4 duties:<\/p>\n Import the BF16 mannequin because the Megatron-Core trainer<\/p>\n Quantize a separate pupil<\/p>\n Distill the scholar towards the frozen trainer<\/p>\n Export the end result<\/p>\n This runs on two nodes with eight GPUs (TP=2, PP=1, EP=4), giving DP=8, with microbatch dimension=1 and world batch dimension=16; practice samples=6,400 yields 400 iterations.<\/p>\n QAD makes it potential to quantize aggressively and nonetheless maintain accuracy near the unique mannequin. On a compact, sparsely activated mannequin akin to Nemotron 3.5 Lightning, that is precisely what unlocks a robust NVFP4 checkpoint whereas preserving accuracy. <\/p>\n The complete QAD recipe for Nemotron 3.5 Lightning is accessible in NVIDIA Mannequin Optimizer. We hope you strive it by yourself fashions and construct on what we’ve got shared right here.<\/p>\n To be taught extra, try the next sources:<\/p>\n We want to thank the Nemotron staff for constructing and open sourcing Nemotron 3.5 Lightning and the post-training datasets that made this work potential, and the NVIDIA Mannequin Optimizer staff for the quantization and distillation tooling that powers this recipe.<\/p>\n We’re grateful to Asma Kuriparambil Thekkumpate, Carlo del Mundo, Chenjie Luo, Daniel Lo, Daria Levy, Frank Solar, Hung-Yueh Chiang, James Shen, Jinhang Choi, Sweta Priyadarshi, Konstantinos Krommydas, Meng Xin, Rohan Joshi, Wei-Ming Chen, Victor Cui, Trenton Starkey, and Yaniv Galronfor their contributions to this work, together with the recipe improvement, ablations, evaluations, and evaluations that formed the outcomes introduced right here.<\/p>\n<\/div>\n
\ndef forward_loop(mannequin):
\n for batch in calib_dataloader:
\n mannequin(batch)<\/p>\n
\n# Instance makes use of W4A16_NVFP4_CFG for quantization
\nmannequin = mtq.quantize(mannequin, mtq.W4A16_NVFP4_CFG, forward_loop=forward_loop)\n<\/p><\/div>\nStep 2: QAD coaching<\/h3>\n
Setting the coaching configuration<\/h4>\n
Quantization scale dealing with throughout QAD<\/h4>\n
![\"Diagram](\"https:\/\/developer-blogs.nvidia.com\/wp-content\/uploads\/2026\/08\/quantization-scale-handling-qad-dynamic-frozen.webp\")
QAD checkpoint evaluations<\/h2>\n
Checkpoint A<\/h3>\n
Checkpoint B<\/h3>\n
Last NVFP4 checkpoint<\/h3>\n
How one can run the QAD recipe with Mannequin Optimizer<\/h2>\n
\nsupply .env-slurm
\nuv run launch.py –yaml examples\/nvidia\/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16\/megatron_lm_qad.yaml –yes\n<\/div>\nStep 1: Create the PTQ pupil <\/h3>\n
\n script: widespread\/megatron_lm\/quantize\/quantize.sh
\n args:
\n – –seq-length 32768 –max-position-embeddings 1048576
\n – –calib-size 32
\n atmosphere:
\n – QUANT_CFG: MAMBA_MOE_NVFP4_CONSERVATIVE_CFG
\n – TP: “1”
\n – EP: “4”\n<\/div>\nStep 2: Distill the scholar from the frozen trainer<\/h3>\n
\n script: widespread\/megatron_lm\/practice\/sft.sh
\n args:
\n – –seq-length 32768 –max-position-embeddings 1048576
\n – –micro-batch-size 1 –global-batch-size 16
\n – –train-samples 6400 # -> 400 iterations
\n – –modelopt-enabled
\n – –export-kd-teacher-load \/cicd\/megatron-lm-bf16\/…\/BF16-MCore
\n – –lr 5.0e-6 –lr-decay-style fixed –lr-warmup-samples 0
\n – –clip-grad 1.0 –weight-decay 0.0
\n – –attention-dropout 0.0 –hidden-dropout 0.0
\n atmosphere:
\n – DATASET: nvidia\/Nemotron-Submit-Coaching-Dataset-v2
\n – TP: “1”
\n – EP: “4”\n<\/div>\nHow one can run the QAD recipe with Megatron-Bridge<\/h3>\n
\nsupply .env-slurm
\nuv run launch.py –yaml examples\/nvidia\/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16\/mbridge_qad.yaml –yes\n<\/div>\nStudy extra<\/h2>\n
Acknowledgments<\/h3>\n