Creating the NVIDIA Nemotron 3 Extremely NVFP4 Checkpoint with NVIDIA Mannequin Optimizer

As context home windows develop longer, shifting massive mannequin weights effectively turns into crucial to efficiency. A typical approach to tackle that is quantization, an optimization approach that compresses mannequin weights right into a smaller knowledge format. One quantization format is NVFP4, an progressive 4-bit floating level launched with NVIDIA Blackwell structure.

That’s the strategy behind our new Nemotron 3 Extremely NVFP4 checkpoint: we quantized the mannequin into NVFP4 utilizing NVIDIA Mannequin Optimizer. The result’s a mannequin that achieves as much as 5.9x larger inference throughput than GLM-5.1 754B FP4 mannequin on decode-heavy workloads whereas matching BF16 accuracy throughout almost each benchmark, as proven in Determine 1.

Whereas the efficiency advantages of NVFP4 are nicely understood, the method of manufacturing a high-quality NVFP4 checkpoint isn’t. This submit walks by means of how we quantized Nemotron 3 Extremely (550B) to NVFP4 with NVIDIA Mannequin Optimizer, and exhibits builders learn how to generate the most effective quantized checkpoints for their very own fashions. 

The Nemotron 3 Extremely NVFP4 checkpoint

A typical false impression is that each layer of an NVFP4 checkpoint is saved in NVFP4. As Desk 1 exhibits, this isn’t the case: completely different layers are quantized to completely different precision codecs, chosen in accordance with every layer’s sensitivity to the structure and its influence on mannequin accuracy. After NVFP4 quantization, the Nemotron 3 Extremely mannequin shrinks from 1,121 GB in BF16 right down to 352.3 GB, a 3.2x discount. The payoff is substantial, chopping the {hardware} footprint in half. 

A key innovation of the Nemotron 3 Extremely NVFP4 is {that a} single checkpoint can run on each NVIDIA Hopper and Blackwell. It achieves this by changing the load format to match the {hardware} it runs on. On Hopper, which lacks native FP4 tensor cores, the serving framework routinely switches to W4A16. On Blackwell, it makes use of native W4A4. 

Whereas W8A8 (8-bit weights, 8-bit activations) looks like the plain Hopper alternative, its bigger reminiscence footprint leaves too little headroom to suit Multi-Token Prediction (MTP). We discovered MTP may solely match alongside W4A16 (4-bit weights, 16-bit activations) so W4A16 matches or beats it throughout the board. Learn the total Nemotron 3 Extremely technical report (Part 4.6) to be taught extra.

How we discovered the optimum NVFP4 checkpoint

Discovering an optimum NVFP4 checkpoint requires some iterations. We dive into the developer story of how we received an NVFP4 checkpoint on this part.

The problem of quantizing at FP4

With FP4 quantization, there are solely 8 optimistic values [0, 0.5, 1, 1.5, 2, 3, 4, and 6] to signify a complete block of weights. We have to decide learn how to map the unique vary of values. That is managed by a scale, primarily a multiplier that determines the granularity of the illustration. Selecting a poor scale means we both waste precision on small values or clip massive values, each of which harm mannequin high quality. So how ought to we select the optimum scale issue? There are a number of approaches.

Max scaling

Right here, we set the size so the biggest worth within the block maps to the utmost representable FP4 worth. Nevertheless, with the presence of a single massive weight outlier, the max scaling compresses each different worth within the block right into a slim vary, which may find yourself flushing these values to zero. This data loss might adversely have an effect on accuracy.  Max scaling preserves the best magnitude worth within the block, with a possible aspect impact of flushing different values to zero.

Strive it with NVIDIA Mannequin Optimizer: 

Max scaling (additionally referred to as absmax, for the reason that scale is ready solely by the block’s absolute most) is the best possibility, however that sensitivity to outliers makes it not often the most effective one. 

That is precisely the hole we hit on our prior mannequin, NVIDIA Nemotron 3 Tremendous: naive absmax NVFP4 PTQ left an accuracy hole, so the staff evaluated a variety of other calibration methods that don’t let a single outlier dictate the size, from imply squared error (MSE)-based weight scaling to GPTQ, an environment friendly methodology that makes use of second-order data to encode weights.

Imply squared error scaling

One other strategy is imply squared error (MSE) scaling, which searches for the size that minimizes common reconstruction error throughout the entire block.

Nevertheless, decrease MSE doesn’t all the time translate to higher mannequin accuracy. MSE calibration lowered per-tensor weight error by 27.1% over four-over-six scaling in our Nemotron 3 Extremely experiments, but produced no constant enchancment on downstream benchmarks.

Strive MSE-based scaling with NVIDIA Mannequin Optimizer: 

For our earlier mannequin, NVIDIA Nemotron 3 Tremendous, the ultimate quantization recipe mixed MSE-based block scaling for weights with a per-tensor FP8 sweep and dynamic max-based scaling for activations. Combining MSE weights with the FP8 activation sweep gave the most effective accuracy-to-size tradeoff of all the things we tried, and it turned our optimum NVFP4 configuration for Tremendous.

Max and MSE scaling each choose a scale to attenuate total rounding error, however neither pays consideration to the place the error comes from on the grid. For Nemotron 3 Extremely, we used a scaling methodology that chooses the vary primarily based on the error from the hole within the grid.

4-over-six scaling

Keep in mind how NVFP4 can solely signify 8 optimistic values: 0, 0.5, 1, 1.5, 2, 3, 4, and 6. Discover that after 4, the subsequent worth jumps straight to six. Any weight that falls in that vary will get rounded aggressively to both 4 or 6, generally incurring over 13% error on a single worth.

4-over-six fixes this as every block of weights independently chooses between scaling to a most of M=4 or M=6, choosing whichever minimizes reconstruction error. 4-over-six works on weights and falls again to the default NVFP4 on activations.

When M=6 wins with block: [2, 4, 5.9, 6] 

At a scale of M=1, values 2, 4, and 6 map precisely onto FP4 grid factors and solely 5.9 rounds to six at negligible value. Scaling to M=4 pushes 2 to 2.25 and 4 to 4.5, introducing error. 

When M=4 wins with block: [10, 20, 30, 40]

Scaling to M=6 maps 30 to 4.62, which rounds right down to 4, a 13% error. Scaling to M=4 as a substitute maps 10, 20, 30, and 40 precisely onto 1, 2, 3, and 4, with zero rounding error throughout your entire block. MSE: 4.33 vs 0.0.

4-over-six was used to set the FP4 routed-expert weight scales in Nemotron 3 Extremely, elevating the worldwide per-tensor weight scale by 1.75x, and with every microblock choosing the M=4 or M=6 grid. Throughout all 49,152 projection weights within the mannequin’s 48 MoE knowledgeable layers, it minimize the median reconstruction MSE by 16.4% in comparison with normal max calibration, and delivered the most effective downstream outcome within the balanced 5.03-BPE setting: 98.5% median restoration relative to BF16, forward of max (96.8%) and MSE (98.4%).

Strive four-over-six with NVIDIA Mannequin Optimizer: 

NVFP4_FOUR_OVER_SIX_CFG might be launched on the upcoming 0.46 NVIDIA Mannequin Optimizer in July. View the Nemotron 3 Extremely PTQ instance. 

Bits-per-element

Efficient bits-per-element (BPE) refers back to the common variety of bits required to retailer all weights of the mannequin. A mannequin with all BF16 weights makes use of 16 efficient bits-per-element, whereas a half-FP8, half-BF16 mannequin makes use of solely 12. NVFP4 provides per-block and per-tensor scaling overhead, bringing its minimal to 4.5 efficient bits-per-element. The per-tensor scale’s 32 bits are amortized throughout the total tensor and is assumed to be negligible within the total BPE calculation.

The objective is to seek for the quantization configuration that pushes efficient BPE as little as potential with out sacrificing accuracy. That is tough as a result of layers should not equally strong. Some are delicate to quantization and should keep in larger precision, which raises the efficient BPE. Since every layer could be quantized at a special stage or left unquantized, the variety of potential combos grows exponentially, making an exhaustive search impractical and a wiser technique essential.

NVIDIA Mannequin Optimizer AutoQuantize (mtq.auto_quantize) does it for you. As a substitute of a set config, you give it a goal bit finances (for instance auto_quantize_bits=4.8) and a listing of candidate codecs, similar to NVFP4_DEFAULT_CFG and FP8_DEFAULT_CFG. It then scores every layer’s sensitivity and searches for the per-layer format project that meets the finances at the most effective accuracy, conserving essentially the most delicate layers within the higher-precision format or skipping them solely.

To search out the precise bits-per-element for Nemotron 3 Extremely, we swept over 5 working factors starting from 4.85 to 7.19 efficient bits-per-element, evaluating accuracy over a number of benchmarks in Desk 3. The important thing sign got here from AA-LCR, the place going from 4.85 to five.03 improved the benchmark by 2.4 factors, and benchmark efficiency then flattened once more past 5.03. This makes 5.03 BPE the candy spot.

How we quantized Nemotron 3 Extremely to NVFP4 with Mannequin Optimizer 

Not like Nemotron 3 Tremendous 120B, Nemotron 3 Extremely is a 550B mannequin, so it advantages considerably from parallelizing the quantization course of. For that reason, we help two quantization paths.

Quantizing Nemotron 3 Extremely to NVFP4 follows the NVIDIA ModelOpt post-training quantization (PTQ) pipeline in NVIDIA Megatron-LM. With the parallel route, the pretrained checkpoint is first transformed to Megatron-LM format after which quantized with a single name to quantize.sh, passing an NVFP4 quantization config because the recipe. On the backend, Megatron-LM shards the mannequin throughout GPUs with knowledgeable and knowledge parallelism (EP = DP = 16 on 16×B300s), so the calibration ahead go runs distributed throughout all units. This reduces load and calibration from ~85 minutes to ~9.

Calibration runs nemotron-post-training-dataset-v2 to suit the per-block scales, and the precision coverage is solely config-driven. Choose it by passing a config to quantize.sh. Both a built-in identify (e.g., NVFP4_DEFAULT_CFG, FP8_DEFAULT_CFG) or a YAML recipe path, which is what in the end will get handed to mtq.quantize(mannequin, config, forward_loop) to put in the quantizers and run calibration.

Strive 4-Over-Six Scaling with NVIDIA Mannequin Optimizer:

NVFP4_FOUR_OVER_SIX_CFG help for four-over-six is touchdown in NVIDIA Mannequin Optimizer 0.46. The Nemotron 3 Extremely recipe for four-over-six is offered on GitHub. 4-over-six works on weights and falls again to default NVFP4 on activations.

Customizing quantization configs

NVIDIA Mannequin Optimizer is constructed to be customizable with completely different quantization configs. The built-in NVFP4 configs vary from NVFP4_DEFAULT_CFG, which quantizes broadly, to extra selective presets like NVFP4_MLP_ONLY_CFG, NVFP4_EXPERTS_ONLY_CFG, and NVFP4_OMLP_ONLY_CFG that limit FP4 to the MLP and knowledgeable layers whereas conserving the delicate consideration projections in larger precision. 

Below the hood, a config is an ordered record of guidelines matched towards module-name patterns, and mtq.quantize() applies them. Weight quantization is ruled by guidelines concentrating on the *weight_quantizer sample, the place you set the format (for NVFP4, E2M1 components with 16-wide blocks and E4M3 block scales), whereas activation quantization is ruled by separate guidelines on the *input_quantizer sample. 

Because the two are impartial, you’ll be able to quantize weights solely or weights and activations collectively, and you may carve out exceptions for particular modules by appending guidelines that disable them. For something past the built-in presets, you’ll be able to write a full YAML recipe and cargo it with –recipe, which then absolutely defines the quant config. 

The next Nemotron-3 Extremely recipe applies NVFP4 with four-over-six to the routed-expert weights, retains the shared specialists and Mamba projections in FP8, makes use of an FP8 KV cache, and leaves all the things else in BF16. The whole recipe ships with NVIDIA Mannequin Optimizer’s recipe library: nvfp4-4o6.yaml

Whereas we walked by means of this on Nemotron 3 Extremely, the identical pipeline works with any Hugging Face mannequin checkpoint. Merely level Mannequin Optimizer at a mannequin card from the Hub or a neighborhood path, choose a config (a built-in preset or your personal recipe), and run the identical quantize and export steps.

Strive the one-click launcher

To simplify deployment, the Mannequin Optimizer launcher automates your entire Extremely PTQ and export workflow. After finishing the setup steps within the launcher README, the workflow could be launched by way of the Nemotron 3 Extremely YAML recipe with a single command from a neighborhood machine:

As soon as launched, the workflow handles the remaining quantization and export steps routinely, assuming entry to a Slurm cluster with ample GPU sources. This instance was validated on 4 nodes, every outfitted with 4 NVIDIA Blackwell GPUs.

For smaller-scale deployments, a PTQ instance can also be obtainable for Nemotron-3 Tremendous:

Get began

This course of could be reproduced utilizing the total recipe obtainable within the open-source NVIDIA Mannequin Optimizer GitHub repository. Mannequin Optimizer is a community-driven undertaking, and contributions are inspired. Points could be filed to report bugs or request options, the undertaking roadmap could be reviewed for upcoming work, and pull requests could be submitted to contribute enhancements. See CONTRIBUTING.md for contribution pointers and getting-started data.

Be taught extra with the next sources:

Acknowledgments

This work wouldn’t have been potential with out the shut collaboration between the NVIDIA Mannequin Optimizer staff and the Nemotron staff. We thank the engineers throughout each groups who contributed to the quantization pipeline, analysis infrastructure, and mannequin coaching. Particular due to the Megatron-LM staff for enabling distributed quantization at scale, and to the Nemotron staff for the benchmark suite used to validate the FP4 recipes. We additionally thank the broader NVIDIA Analysis and Utilized Deep Studying groups for his or her continued help and suggestions all through this undertaking.

Particularly, we thank Asma Kuriparambil Thekkumpate, Jenny Chen, and Jinhang Choi for main the implementation of the NVFP4 quantization on Nemotron 3 Extremely.