Pre-training frontier LLMs comes right down to throughput. When coaching spans trillions of tokens throughout 1000’s of accelerators, each share level of step time can add as much as days of coaching and substantial compute prices. Numerical precision is without doubt one of the highest-leverage knobs obtainable, however low- bit mixed-precision pretraining is tough to get proper.
To handle this, the NVFP4 coaching recipe in TransformerEngine makes use of subbyte precision for JAX pretraining. For an end-to-end instance, see the recipe in MaxText, a high-performance, scalable LLM framework library. The result’s high-throughput, 4-bit mixed-precision pre-training on NVIDIA Blackwell with no measurable accuracy loss in comparison with the FP8 baseline.
This publish explains the NVFP4 format and the way it’s constructed to realize excessive efficiency and accuracy at ultra-low precision. It additionally exhibits easy methods to apply a MaxText NVFP4 pretraining recipe and gather efficiency knowledge displaying efficiency positive aspects. For methodology particulars, see the NVFP4 pretraining paper.
NVFP4 format and advantages
This NVFP4 introductory publish explains its format and the way two-level microscaling encodes increased alerts with much less error than different microscaling codecs. It additionally explains how native {hardware} assist of NVFP4 on the NVIDIA GB300 Grace Blackwell Extremely Superchip delivers 7x GEMM throughput in comparison with native FP8 precision on the NVIDIA Hopper. That increased throughput, together with the NVFP4 pretraining recipe, shortens coaching step time with negligible accuracy loss. This permits AI factories to coach extra and bigger fashions inside the similar time finances, or prepare fashions sooner with a shorter time finances.
NVFP4 pretraining recipe
The NVFP4 recipe combines a number of components that collectively protect convergence whereas unlocking NVIDIA Blackwell and the NVIDIA Rubin platform NVFP4 throughput. To allow environment friendly narrow-precision coaching, the pretraining recipe makes use of a number of key methods which have been chosen primarily based on their efficiency and accuracy.
5 key components work collectively whereas sustaining the accuracy required in 4-bit pretraining:
Micro block scaling makes use of 16-element blocks, half the scale of MXFP4‘s 32-element blocks, so a single outlier has much less affect on the shared scale.
E4M3 block scale components makes use of mantissa bits as a substitute of MXFP4’s power-of-two E8M0 scaling, layered beneath a per-tensor FP32 scale. In an 8B-parameter, 1T token experiment, MXFP4 requires ~36% extra tokens to match NVFP4’s remaining loss.
Random Hadamard Rework applies solely to WGRAD GEMM inputs to Gaussianize outliers. The recipe skips on FPROP and DGRAD as a result of reworking these paths would additionally require reworking the load, breaking 2D-scale consistency.
2D weight scaling makes use of one FP8 scale per 16×16 weight block, so FPROP and its transposed DGRAD use the identical scale. Activation and gradients maintain lower-overhead 1×16 scaling.
Stochastic rounding makes use of unbiased rounding to maintain tiny updates from being crushed to zero. Weights and activations keep on round-to-nearest-even, the place SR would amplify error as a substitute. Each modes are native to Blackwell FP4 conversion directions.
Determine 1 exhibits the NVFP4 knowledge circulate inside one linear layer.
The three GEMMs: FPROP (ahead), DGRAD (activation gradient), and WGRAD (weight gradient) are quantized to NVFP4 just for the MLP (feed-forward) layers of the transformer; the GEMMs inside the eye block (QKV projection, consideration output projection, and the rating/context matmuls) stay in increased precision.
NVFP4 is utilized first to MLP layers as a result of consideration’s softmax exponentially amplifies quantization noise on QK^T scores. Consideration activations additionally carry concentrated outliers that 4-bit precision can’t signify properly. As a result of MLPs account for many coaching FLOPs, this captures a lot of the speedup with out risking convergence.
All three MLP GEMMs eat NVFP4 inputs and emit BF16 outputs, that are ultimately folded into an FP32 grasp weight on the optimizer step. The identical path makes the recipe’s convergence‑preserving decisions seen: 2D block quantization on the weights (constant FPROP/DGRAD values throughout the transpose), a Random Hadamard Rework on the WGRAD inputs (flattens outliers earlier than 4-bit quantization), and stochastic rounding on the gradient quantizers (retains small updates unbiased).
Enabling NVFP4 in MaxText
The MaxText NVFP4 recipe is obtainable within the JAX-Toolbox GitHub repository. The launch script trains Llama 3 8B with NVFP4 on Blackwell. To allow it, set the quantization flag in MaxText to change into the NVFP4 path. Two modes are uncovered:
quantization=te_nvfp4: NVFP4 with Random Hadamard Rework. Really helpful when the convergence beneath te_nvfp4_no_rht isn’t passable.
quantization=te_nvfp4_no_rht: NVFP4 with out RHT. Lowest overhead, however might degrade convergence high quality.
Run the instance script from the MaxText repository root inside a container that has JAX, NVIDIA Transformer Engine, and the required NVIDIA CUDA/cuDNN libraries put in. The general public NVIDIA MaxText container ghcr.io/nvidia/jax:maxtext is really useful.
The next is a partial instance of the Llama3 8B MaxText NVFP4 coaching script, which declares the nvfp4 argument by Transformer Engine:
After launch, MaxText prints step time, TFLOP/s/machine, and tokens/s/machine. An NVIDIA Nsight Programs hint is written to base_output_directory for inspection. To provide the FP8 baseline used within the comparability beneath, run the identical script with quantization=te_fp8_delayedscaling.
Efficiency outcomes
The benchmark makes use of MaxText pre-training on Llama 3 8B with FSDP=4 with a sequence size of 8,192, a per-device batch dimension of 4, and 50 steps inside the general public ghcr.io/nvidia/jax:maxtext container.
Desk 1 summarizes MaxText pretraining efficiency on the NVIDIA GB200 Grace Blackwell Superchip and NVIDIA GB300 Grace Blackwell Extremely Superchip for Llama 3 8B and Llama 3.1 405B, evaluating the NVFP4 recipe in opposition to an FP8 baseline on the identical {hardware}, parallelism, and world batch dimension. Numbers are measured at sequence size 8,192.
Determine 2 exhibits per‑GPU sustained TFLOP/s throughout the 4 baseline configurations. NVFP4 delivers a further 500–700 TF/s per GPU on each configuration. The 1.31–1.73x speedup over the FP8 baseline is from altering the GEMM precision whereas holding the mannequin, hyperparameters, parallelism, and world batch dimension an identical.
The most important relative positive aspects are with the 405B configurations (1.44x on GB200, 1.73x on GB300), the place the per‑step GEMM mass dominates FSDP collective overhead and a precision‑degree speedup interprets instantly into wall‑clock financial savings.
Determine 3 overlays Llama 3 8B coaching loss for the FP8 baseline and NVFP4 throughout 10,000 pretraining steps with in any other case an identical hyperparameters. Each runs descend the identical curve from ≈12.2 nats to ≈3.9 nats, with a converged‑regime imply hole of simply +0.026 nats, properly inside step‑to‑step noise. The NVFP4 speedups in Determine 2 include no measurable accuracy price.
Get began
Pull the MaxText container, run nvfp4_example.sh on Blackwell to get began.
Acknowledgments
For his or her contributions to NVFP4 enablement in JAX, XLA, and TE, particular because of Jaroslav Sevcik, Ilia Sergachev, Johannes Reifferscheid, Phuong Nguyen, and Jeremy Berchtold.
