Hugging Face integrates Nunchaku 4-bit diffusion inference into Diffusers

Hugging Face has integrated Nunchaku’s 4-bit diffusion inference into Diffusers via a new feature called Nunchaku Lite, enabling native loading of quantized models without custom pipelines or local CUDA compilation. This integration leverages the SVDQuant quantization method, which uses 4-bit weights and activations (W4A4) to reduce memory usage and accelerate the denoising loop.

The Nunchaku Lite approach differs from traditional weight-only quantization methods, such as bitsandbytes or GGUF, which store weights in low precision but dequantize them at compute time. While these methods reduce memory usage, they often do not improve inference speed and can introduce latency overhead. SVDQuant, by contrast, runs the main transformer layers with 4-bit weights and activations, delivering both memory savings and speed improvements.

Pre-quantized checkpoints are now available on the Hugging Face Hub, allowing users to load and run models with a single call to `from_pretrained()`. For example, the ErnieImagePipeline checkpoint `lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder` generates a 1024x1024 image in about 1.7 seconds on an RTX 5090 with a peak memory usage of ~12 GB, compared to ~24 GB for the BF16 pipeline.

Nunchaku Lite supports two kernel families: `svdq_w4a4` for 4-bit weights and activations with a low-rank correction, and `awq_w4a16` for 4-bit weights with 16-bit activations. The former is used for compute-intensive layers like attention and MLP projections, while the latter is applied to memory-bound and precision-sensitive layers such as adaptive normalization modules. While Nunchaku Lite does not match the speedup of the original Nunchaku engine due to the lack of architecture-specific fused kernels, it still delivers around a 30% speedup while retaining the same level of VRAM reduction.

Hardware support is segmented by precision and GPU generation. The `svdq_w4a4` kernel with NVFP4 precision is supported on NVIDIA Blackwell GPUs (e.g., RTX 50 series, RTX PRO 6000, B200), while earlier generations require INT4 variants. A hardware support table in the announcement outlines compatibility for each scheme and precision.

To use Nunchaku Lite, users install updated versions of Diffusers, Transformers, Accelerate, and the Hugging Face kernels package. The integration is designed to feel familiar to users of existing quantization backends in Diffusers, with the only special requirement being a `quantization_config` block in the model’s `config.json` that specifies the quantization method, compute dtype, and targeted modules.

Hugging Face also introduced a companion toolkit, diffuse-compressor, which allows users to quantize new architectures and publish them as standard Diffusers repositories. This toolkit supports inspecting models for quantization, running the quantization process, packaging a Diffusers pipeline, and pushing the quantized model to the Hub for verification and sharing.