The right way to Optimize Transformer-Primarily based Fashions for Low-Precision Coaching

Transformer architectures are the spine of many fashionable giant language and generative AI fashions. As these fashions develop in measurement, coaching runs devour extra GPU hours and extra engineering iteration time. Accelerating transformers is due to this fact not only a efficiency optimization, however straight impacts how rapidly groups can experiment and the way giant a mannequin they’ll afford to coach. NVIDIA Hopper and NVIDIA Blackwell GPUs assist resolve this downside by introducing low-precision operator assist together with FP8 and NVFP4.

Transformers spend a lot of their coaching time in GEMMs, and low-precision codecs pace up coaching primarily by making these matrix multiplications sooner and cheaper. Nevertheless, your transformer config doesn’t let you know which GEMMs are literally working in your mannequin. If you wish to perceive the place coaching time goes, you want to flip your transformer config and batch measurement into the precise M×Okay×N matrix shapes your mannequin executes, then benchmark these shapes throughout precisions. This may show you how to decide the optimum precision to your structure earlier than committing to a costlier coaching run.

NVIDIA Transformer Engine (TE) can deal with quantization and kernel dispatch unlocking low precision codecs. This publish exhibits you the right way to transfer from high-level mannequin settings to concrete GEMM workloads, profile them with a microbenchmark, and estimate the place decrease precision will really translate into speedups that can assist you speed up your transformer-based fashions. The use case options CodonFM, a language mannequin for biology centered on RNA.  

Mannequin configuration and coaching inputs

Suppose you’re working with a 5B-parameter mannequin akin to CodonFM 5B. It is going to have a config akin to:

Your coaching configuration is:

The benchmark instrument can then take these hyperparameters straight after which use a single command to derive GEMM shapes, benchmark them throughout precisions, and compute the complete speedup evaluation:

Notice: To disable Blackwell-specific flags, add –no-fp8 –no-fp4. –no-fp8 –no-fp4 gives BF16 plus the three FP8 recipes that work on Hopper.

–no-fp8 disables MXFP8 

–no-fp4 disables NVFP4

Utilizing autocast mode versus prequantizing

By default, the instrument runs in autocast mode, which is what TE does throughout coaching: inputs are dynamically quantized to the goal precision earlier than every GEMM, so the measured time contains each the quantization value and the GEMM kernel itself. This gives you with the life like per-GEMM image throughout a coaching step.

The instrument computes M = 31 × 512 = 15,872 tokens, derives all 12 GEMM shapes, benchmarks every throughout enabled precisions, and prints the complete outcomes. Fprop, Dgrad, and Wgrad shapes are all benchmarked individually to seize the affect of various matrix side ratios on kernel choice. 

By default, the instrument runs in autocast mode, which is what TE does throughout coaching: inputs are dynamically quantized to the goal precision earlier than every GEMM, so the measured time contains each the quantization value and the GEMM kernel itself. This gives you with the life like per-GEMM image throughout a coaching step.

The instrument computes M = 31 × 512 = 15,872 tokens, derives all 12 GEMM shapes, benchmarks every throughout enabled precisions, and prints the complete outcomes. Fprop, Dgrad, and Wgrad shapes are all benchmarked individually to seize the affect of various matrix side ratios on kernel choice.

To isolate uncooked GEMM kernel efficiency, add –pre-quantize. This prequantizes all inputs as soon as earlier than the timed loop, so the measured time displays solely the GEMM kernel execution—no dynamic quantization, no block scaling computation, no format conversion through the timed area.

Notice that FP8 DelayedScaling at all times runs in autocast mode, even with –pre-quantize as a result of it depends on an amax historical past that requires dynamic quantization. Its occasions are due to this fact circuitously akin to different precisions in prequantized mode.

Evaluating the autocast and prequantized speedups tells you precisely how a lot quantization overhead prices: NVFP4 versus BF16 goes from 1.98x (autocast) to three.48x (kernel-only). The hole between these two numbers is the overhead from dynamic quantization, Hadamard transforms, and block scaling that happens in every coaching step.

Use autocast outcomes for predicting actual coaching speedups. That is what TE really does throughout coaching. Use prequantized outcomes to grasp whether or not quantization overhead is the bottleneck, or to check uncooked tensor core throughput throughout precisions unbiased of the quantization implementation.

Decoding the outcomes for an actual mannequin

This part walks via the right way to interpret these outcomes for an actual mannequin. Utilizing the identical CodonFM 5B config, we ran the complete mannequin config benchmark on NVIDIA B300. The per-shape NVFP4 versus MXFP8 speedups from the Fprop outcomes are as follows:

Pay attention to the next factors: 

The eye output GEMM receives minimal profit from decrease precision. In contrast with the MXFP8 baseline, there’s solely a 1.05x speedup. That is the smallest weight matrix within the layer (4096×4096)—barely giant sufficient for decrease precision to beat the overhead. Against this, the a lot bigger MLP Down GEMM delivers 1.66x NVFP4 over MXFP8 on the identical {hardware}. The MLP down GEMM is large enough to amortize the quantization overhead, the place consideration output isn’t.

The large GEMMs present actual however subtheoretical good points. The FP4 tensor cores ship 1.46x to 1.66x over MXFP8 on the massive GEMMs. That is properly in need of the theoretical 2x to 3x from the {hardware} spec. When you embrace the eye output GEMM, the blended Fprop speedup drops to 1.47x. After including Wgrad occasions, non-GEMM overhead and NVFP4-specific quantization prices, the end-to-end hole between NVFP4 and MXFP8 in coaching is in keeping with these kernel-level numbers.

FP8 DelayedScaling is surprisingly aggressive on NVIDIA Blackwell. At 7.80 ms/layer in autocast mode, it outperforms each FP8 CurrentScaling (9.15 ms) and MXFP8 (8.98 ms). In prequantized mode FP8 CurrentScaling pulls forward (6.81 ms versus 8.12 ms), suggesting the DelayedScaling amax-history strategy has decrease quantization overhead however related uncooked kernel throughput. This can be a good instance of the comparability between autocast and prequantized surfacing completely different winners relying on whether or not you measure with or with out the quantization tax.

The prequantized outcomes reveal the true kernel potential. Working with –pre-quantize removes quantization overhead totally, and NVFP4 versus BF16 jumps from 1.98x (autocast) to three.48x (kernel-only). This exhibits the FP4 tensor cores are delivering actual speedups. It’s the quantization overhead in autocast mode that narrows the hole.

The Fprop versus Dgrad comparability reveals that the 2x approximation is imprecise for quantized codecs. Whereas BF16 Dgrad is inside 2% of Fprop, quantized codecs present 5–13% slower Dgrad sums. The QKV Proj Dgrad is very uneven—33–51% slower than Fprop for FP8/FP4—as a result of swapping Okay (4096) and N (12288) dramatically adjustments the matrix side ratio and kernel choice. That is precisely why the instrument benchmarks Fprop and Dgrad individually relatively than counting Fprop time twice.

Upon getting the estimated GEMM-only speedup, evaluate it towards your noticed end-to-end coaching speedup:

GEMM speedup ≈ coaching speedup: GEMMs dominate the step, the whole lot is working as anticipated

GEMM speedup >> coaching speedup: Overhead outdoors of GEMMs is consuming the good points. For NVFP4 specifically, this overhead contains Random Hadamard transforms on Wgrad inputs, stochastic rounding on gradients, 2D block scaling for weights, and the additional reminiscence cross for per-tensor amax computation. These are all further ops that MXFP8 doesn’t want, they usually can considerably slim the hole even when the uncooked FP4 GEMMs are a lot sooner

GEMM speedup ≈ 1.0 even within the microbenchmark. The FP4 kernels aren’t really sooner at these shapes, or they’re silently falling again to FP8

The final case is very value checking. Set NVTE_LOG_LEVEL=1 or examine with NVIDIA Nsight Techniques to verify that TE is definitely dispatching FP4 kernels. TE can silently fall again to FP8 or BF16 for layers or ops that don’t assist FP4 but, which might clarify equivalent efficiency with no different signs. You too can evaluate GPU reminiscence utilization between MXFP8 and NVFP4 runs. If reminiscence is sort of equivalent, that’s a robust sign that FP4 weights aren’t really being saved.

Get began benchmarking your mannequin for low-precision coaching 

Low-precision coaching speedups are extremely depending on the precise GEMM shapes your mannequin runs and working in low precision doesn’t routinely translate into end-to-end coaching good points, particularly when quantization overhead, kernel choice, and non-GEMM operations are included. By turning a transformer config into concrete M×Okay×N workloads, you may benchmark BF16, MXFP8, and NVFP4 on the shapes that matter to your mannequin earlier than committing to a full coaching run.

Benchmark your GEMMs to see which precision is best for you. To get began, take a look at the benchmark script. For the complete documentation and to grasp how these shapes are derived, see the GEMM profiling tutorial within the Transformer Engine documentation.

Use this benchmark to:

Autocast outcomes to set life like training-speedup expectations

Prequantize outcomes to know whether or not you’re bottlenecked on kernels or on quantization

Run candidate mannequin configs via the instrument earlier than committing to a coaching run, because the instrument is a helpful structure co-design instrument