[2603.10444] The Curse and Blessing of Imply Bias in FP4-Quantized LLM Coaching

View a PDF of the paper titled The Curse and Blessing of Imply Bias in FP4-Quantized LLM Coaching, by Hengjie Cao and 17 different authors

View PDF
HTML (experimental)

Summary:FP4 coaching guarantees substantial reminiscence and compute financial savings for big language fashions, however stays fragile as a result of blockwise quantization is dictated by excessive activation magnitudes, which inflate dynamic vary and compress long-tail indicators. We establish a counterintuitive supply of this failure: dominant activation outliers usually are not merely arbitrary sparse occasions, however are largely induced by a coherent rank-one imply bias, whose route aligns with the main anisotropic spectral element. This imply element strengthens throughout coaching, is amplified and reshaped by consideration and FFN operators, and more and more dominates high activation magnitudes. Crucially, this discovery reveals {that a} seemingly advanced outlier-suppression drawback admits a very easy answer: isolate the coherent imply earlier than quantization. We due to this fact suggest Averis, a mean-residual splitting quantization methodology that separates the imply element utilizing solely reductions and elementwise subtractions earlier than FP4 quantization. Throughout Qwen3 0.6B Dense skilled on 100B tokens and Qwen3 7B A1.5B MoE skilled on 50B tokens, Averis permits strong W4A4G4 FP4 coaching, lowering BF16 loss gaps to 1.19%/0.81% versus 2.05%/1.10% for NVIDIA’s lately launched Hadamard-based outlier-smoothing methodology, whereas limiting downstream gaps to 0.89/0.71 factors. With solely 2.20% end-to-end overhead over vanilla NVFP4, about 30% of NVIDIA’s Hadamard-based design, Averis supplies a hardware-efficient path to secure low-bit LLM coaching. Complementary to Hadamard, Averis additional reduces the Qwen3-0.6B loss and downstream gaps to 0.94% and 0.73 factors when mixed. Code is offered at: this https URL.