Massive diffusion transformers can create gorgeous photographs (and even movies, audio snippets, and now textual content), however loading a contemporary text-to-image mannequin in BF16 precision typically requires 20-30 GB of VRAM, which places these fashions out of attain of most client GPUs. Quantization is a robust resolution to this downside, and Diffusers already integrates a number of quantization backends reminiscent of bitsandbytes, GGUF, torchao, and Quanto, which we lined in Exploring Quantization Backends in Diffusers.<\/p>\n
Most of those backends are weight-only. Because of this they retailer the weights in low precision and dequantize them again to excessive precision at compute time. This reduces reminiscence utilization considerably, nevertheless it normally doesn’t make inference sooner, and might even add a small latency overhead.<\/p>\n
SVDQuant, the quantization methodology behind the favored Nunchaku inference engine, takes a distinct strategy. It runs the principle transformer layers with 4-bit weights and activations (W4A4), decreasing reminiscence whereas additionally dashing up the denoising loop. The small print are lined under, however till now, utilizing these checkpoints required a separate inference library.<\/p>\n
With present Diffusers, loading a Nunchaku checkpoint is so simple as calling from_pretrained(), with no native CUDA compilation required due to the kernels package deal. As well as, the companion diffuse-compressor toolkit helps you to quantize new architectures your self and publish them as common Diffusers repositories.<\/p>\n\n \n<\/figure>\n
\n
\t \n\t\tDesk of Contents \n\t<\/span> \n<\/h2>\n
\n
\t \n\t\tGetting began with Nunchaku Lite \n\t<\/span> \n<\/h2>\n
First, set up the necessities. You want a latest model of Diffusers and the Hugging Face kernels package deal:<\/p>\n
pip set up -U diffusers transformers speed up kernels bitsandbytes<\/p>\n
Then load a pre-quantized pipeline like every other Diffusers mannequin:<\/p>\n
picture = pipe( \n immediate=“A cinematic portrait of a crimson fox in a misty forest at dawn, “<\/span> \n “detailed fur, volumetric mild”<\/span>, \n top=1024<\/span>, \n width=1024<\/span>, \n num_inference_steps=8<\/span>, \n guidance_scale=1.0<\/span>, \n generator=torch.Generator(“cuda”<\/span>).manual_seed(42<\/span>), \n).photographs[0<\/span>] \npicture.save(“output.png”<\/span>)<\/p>\n\n \n<\/figure>\n
No customized pipeline class or separate inference engine is required, and there may be nothing to compile domestically. The NVFP4 kernels are downloaded from the Hub via the Nunchaku Lite kernels web page the primary time they’re used. This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 textual content encoder, and generates a 1024×1024 picture in about 1.7 seconds on an RTX 5090 with a peak reminiscence utilization of about 12 GB, in contrast with about 24 GB for the BF16 pipeline. You will discover extra particulars concerning the Nunchaku Lite checkpoint format within the official Diffusers documentation.<\/p>\n
\n
NVFP4 checkpoints require an NVIDIA Blackwell GPU (RTX 50 sequence, RTX PRO 6000, B200). For earlier generations, use the INT4 variants. See the {hardware} help desk under for particulars.<\/p>\n<\/blockquote>\n
\n
\t \n\t\tBackground: SVDQuant and Nunchaku \n\t<\/span> \n<\/h2>\n
SVDQuant is the quantization methodology behind Nunchaku, its reference CUDA inference engine. Normal 4-bit quantization is troublesome for diffusion transformers as a result of each weights and activations comprise massive outliers. SVDQuant handles this by transferring activation outliers into the weights, representing the toughest a part of every weight matrix with a small 16-bit low-rank department, and quantizing the remaining residual to 4 bits. Nunchaku makes this quick with fused kernels for the 4-bit path and the low-rank department.<\/p>\n\n Nunchaku fuses the low-rank down projection with the quantization kernel and the low-rank up projection with the 4-bit compute kernel, eliminating the reminiscence entry overhead of the 16-bit department. Determine from the SVDQuant paper.<\/figcaption><\/figure>\n
\n
\t \n\t\tIntroducing Nunchaku Lite \n\t<\/span> \n<\/h2>\n
The unique Nunchaku engine will get a lot of its pace from model-specific fused execution paths, reminiscent of fused QKV projections and fused GELU\/MLP kernels. These optimizations are tied to every structure’s module format and checkpoint format, so supporting a brand new mannequin household normally requires model-specific integration work.<\/p>\n
Nunchaku Lite is the brand new integration path in Diffusers. With it, Diffusers can load Nunchaku-style checkpoints with no customized pipeline or a separate inference engine. Below the hood, Nunchaku Lite patches the related nn.Linear modules of a inventory Diffusers mannequin with runtime SVDQ\/AWQ linear layers earlier than the checkpoint is loaded. The CUDA kernels come from the Hub via the kernels package deal. Two kernel households are used:<\/p>\n
svdq_w4a4: 4-bit weights and activations with the SVDQuant low-rank correction. This layer is used for the transformer’s consideration and MLP projections, the place practically the entire compute is spent, and is accessible in INT4 and NVFP4 variants. \nawq_w4a16: 4-bit weights with 16-bit activations, used for adaptive normalization and modulation projections reminiscent of FLUX adanorm_single \/ adanorm_zero or Qwen-Picture modulation layers. These layers are memory-bound and precision-sensitive, making AWQ an excellent match to protect precision whereas nonetheless saving reminiscence and house.<\/p>\n
The trade-off is that, with out architecture-specific fused kernels and modules, Nunchaku Lite can’t match the speedup of the unique Nunchaku engine. Nonetheless, the bare-bones implementation nonetheless delivers round 30% speedup whereas retaining the identical stage of VRAM discount.<\/p>\n
\n
\t \n\t\tNative loading in Diffusers \n\t<\/span> \n<\/h2>\n
When you have used bitsandbytes or torchao in Diffusers, the mechanics will really feel acquainted. A Nunchaku Lite mannequin repository is an peculiar Diffusers repository. The one particular half is a quantization_config block contained in the transformer’s config.json:<\/p>\n
This config tells Diffusers which modules have been quantized, which scheme they use, and which Nunchaku Lite runtime layer to instantiate (SVDQW4A4Linear or AWQW4A16Linear).<\/p>\n
As a result of the quantized mannequin retains the precise module construction of the dense one, the whole lot downstream (schedulers, LoRA loading hooks, offloading, torch.compile) sees a traditional Diffusers mannequin.<\/p>\n
\n
\t \n\t\t{Hardware} help \n\t<\/span> \n<\/h3>\n
Nunchaku Lite makes use of completely different kernel variants relying on the GPU era and checkpoint precision:<\/p>\n
\n
Scheme \nPrecision \nSupported GPUs<\/p>\n
svdq_w4a4 \nnvfp4 \nBlackwell (RTX 50 sequence, RTX PRO 6000, B200)<\/p>\n
Volta and Hopper GPUs are presently not supported by the 4-bit kernels. The quantizer validates the GPU’s CUDA functionality at load time and raises a transparent error as an alternative of manufacturing incorrect outputs.<\/p>\n<\/blockquote>\n
\n
\t \n\t\tGetting extra pace and decrease reminiscence \n\t<\/span> \n<\/h2>\n
Nunchaku Lite could be mixed with different Diffusers reminiscence and pace optimizations.<\/p>\n
torch.compile. Compiling the transformer improves the end-to-end speedup from 1.35x to 1.8x:<\/p>\n
Quantized textual content encoders. The transformer will not be the one part with a big reminiscence footprint. Textual content encoders reminiscent of T5 or Qwen3 can occupy a number of gigabytes on their very own. Additional quantizing the textual content encoder with bitsandbytes NF4 reduces peak VRAM by about 22% in our benchmark.<\/p>\n
Offloading. Diffusers offloading helpers reminiscent of enable_model_cpu_offload() and enable_sequential_cpu_offload() work as ordinary if it’s essential match the pipeline onto a smaller GPU.<\/p>\n
\n
\t \n\t\tBenchmarks \n\t<\/span> \n<\/h2>\n
All numbers under have been measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024×1024 utilizing rootonchair\/ERNIE-Picture-Turbo-nunchaku-lite-int4-bnb4-text-encoder.<\/p>\n
\n
\t \n\t\tFinish-to-end latency and reminiscence \n\t<\/span> \n<\/h3>\n
BF16 baseline \n3.00 s \n2.86 s \n31.1 GB \n1.0x<\/p>\n
Nunchaku Lite NVFP4 \n2.27 s \n2.13 s \n20.6 GB \n1.35x<\/p>\n
Nunchaku Lite NVFP4 + torch.compile \n1.68 s \n1.53 s \n20.6 GB \n1.8x<\/p>\n
Nunchaku Lite NVFP4 + NF4 textual content encoder \n2.29 s \n2.13 s \n16.0 GB \n1.35x<\/p>\n<\/div>\n
As proven above, Nunchaku reduces peak VRAM by as much as 50% whereas nonetheless bettering latency by roughly 30%. The remaining overhead comes largely from additional kernel launches, which torch.compile can mitigate, bringing the complete pipeline right down to 1.68 s, or 1.8x sooner than the BF16 baseline.<\/p>\n
\n
\t \n\t\tPicture high quality \n\t<\/span> \n<\/h3>\n\n BF16 vs 4-bit outputs with equivalent seeds and settings.<\/figcaption><\/figure>\n
\n
\t \n\t\tQuantizing your individual mannequin \n\t<\/span> \n<\/h2>\n
Nunchaku Lite help in Diffusers is architecture-agnostic, and the diffuse-compressor toolkit offers an end-to-end SVDQuant workflow for Diffusers fashions: calibrate, quantize, package deal, and publish.<\/p>\n
Beneath, we stroll via quantizing FLUX.2 Klein 4B for instance. It covers the principle steps: examine the mannequin, calibrate and quantize the transformer, package deal the end result as a Diffusers pipeline, then confirm and push it to the Hub. The complete tutorial covers each flag intimately.<\/p>\n
\n
\t \n\t\t1. Examine what will probably be quantized \n\t<\/span> \n<\/h3>\n
The generic scanner walks the mannequin and decides what to focus on: appropriate linears contained in the repeated transformer-block stack turn out to be SVDQ W4A4 targets, acknowledged modulation linears turn out to be AWQ W4A16 targets, and the whole lot else stays dense.<\/p>\n
All the time learn this report earlier than quantizing. For FLUX.2 Klein 4B, the anticipated result’s 100 SVDQ targets, 3 AWQ targets, and 6 dense outer linears, with no lacking patterns or duplicate names.<\/p>\n
\n
\t \n\t\t2. Run quantization \n\t<\/span> \n<\/h3>\n
The next command runs SVDQuant on the transformer and writes the quantized checkpoint to outputs\/checkpoints\/svdq-int4_r32-flux-2-klein-4b.safetensors:<\/p>\n
Substitute –precision int4 with nvfp4 to construct Blackwell-native weights.<\/p>\n
\n
\t \n\t\t3. Package deal a Diffusers pipeline \n\t<\/span> \n<\/h3>\n
The converter combines the quantized transformer with the bottom pipeline’s different elements, writes the compact nunchaku_lite configuration into transformer\/config.json, and might optionally convert textual content encoders to NF4:<\/p>\n
As soon as the outputs look good, run pipe.push_to_hub(“your-name\/your-model-nunchaku-lite-int4”). Different customers can then load it with the identical from_pretrained() sample proven above.<\/p>\n
\n
\t \n\t\tQuantizing fashions with structural rewrites \n\t<\/span> \n<\/h3>\n
Be aware that the generic path assumes the structure could be quantized with out structural rewrites. For added speedup, the unique Nunchaku engine rewrites teams of Diffusers layers as fused modules. The generic path can’t infer these adjustments by itself, reminiscent of combining separate Q, Ok, and V projections into one module or splitting a fused projection throughout a number of modules.<\/p>\n
FLUX.1-dev’s QKV projection is a concrete instance. Diffusers defines three separate modules:<\/p>\n
This grouped module is required as a result of Nunchaku’s fused operator consumes the QKV projection, Q\/Ok normalization, and rotary embeddings collectively. By comparability, the default Diffusers path executes them individually:<\/p>\n
That is the structural rewrite that the generic path can’t infer. Diffusers has three vacation spot modules with to_q, to_k, and to_v parameter prefixes, whereas Nunchaku has one grouped module beneath to_qkv. A model-specific goal config or adapter should state that the Q, Ok, and V parameters needs to be concatenated alongside the output dimension, in that order, and loaded into to_qkv.<\/p>\n
Structural rewrites like these are described by a model-specific goal config throughout quantization and dealt with by a small runtime adapter when the checkpoint is loaded. \nThe FLUX.2 Klein 4B quantization script offers a concrete target-config instance for producing a structurally rewritten checkpoint, whereas rootonchair\/nunchaku-lite offers the runtime adapters wanted to load grouped QKV tensors, cut up fused projections, and different fused operations. \nFor the whole workflow, you’ll be able to verify the Including A New Mannequin information.<\/p>\n
To get began instantly, take a look at the next repositories:<\/p>\n
\n
\t \n\t\tConclusion \n\t<\/span> \n<\/h2>\n
Nunchaku’s SVDQuant kernels are some of the efficient methods to run diffusion transformers effectively on client {hardware}, and they’re now natively supported in Diffusers. Pre-quantized checkpoints load with from_pretrained(), and the diffuse-compressor toolkit makes it attainable to quantize new architectures with out ready for engine help. By quantizing each weights and activations, the W4A4 path lowers reminiscence use whereas bettering denoising latency, protecting picture high quality near the BF16 authentic.<\/p>\n
If you happen to quantize and publish a brand new mannequin, we might love to listen to about it. Share it on the Hub and tell us! When you have any questions on this characteristic, be happy to affix our Discord.<\/p>\n
To study extra, take a look at the next assets:<\/p>\n
\n
\t \n\t\tAcknowledgements \n\t<\/span> \n<\/h2>\n
Due to the Diffusers maintainers for evaluations and steering all through the combination, and to the MIT HAN Lab \/ Nunchaku staff for the unique SVDQuant work. Due to Marc Solar for offering suggestions on the weblog publish. Due to \u00c1lvaro Somoza for attempting out nunchaku-lite and for offering suggestions.<\/p>\n
rootonchair can be grateful to SilverAI for supporting this work and offering the setting through which a lot of this improvement passed off.<\/p>\n<\/div>\n
![\"Pham](\"https:\/\/cdn-avatars.huggingface.co\/v1\/production\/uploads\/655647bf2f76548766fce60f\/mZhWuhJb-bzlzp-C3VoO_.jpeg\"\/)
<\/span> <\/span> <\/div>\n![\"Sayak](\"https:\/\/cdn-avatars.huggingface.co\/v1\/production\/uploads\/1649681653581-5f7fbd813e94f16a85448745.jpeg\"\/)
<\/span> <\/span> <\/div>\n<\/div><\/div>\n<\/div>\n<\/div>\n![\"Nunchaku](\"https:\/\/huggingface.co\/datasets\/huggingface\/documentation-images\/resolve\/main\/blog\/nunchaku-diffusers\/contact_sheet_top3_metrics_bold.png\")
![\"BF16](\"https:\/\/huggingface.co\/datasets\/huggingface\/documentation-images\/resolve\/main\/blog\/nunchaku-diffusers\/fox_bf16_vs_nunchaku_no_metrics.png\")
![\"Nunchaku](\"https:\/\/huggingface.co\/datasets\/huggingface\/documentation-images\/resolve\/main\/blog\/nunchaku-diffusers\/svdquant_kernel_fusion.png\")
![\"Quality](\"https:\/\/huggingface.co\/datasets\/huggingface\/documentation-images\/resolve\/main\/blog\/nunchaku-diffusers\/quality_grid.png\")