Hugging Face Diffusers integration with NVIDIA NeMo Automodel enables distributed fine-tuning of diffusion models

NVIDIA and Hugging Face announced an open-source integration that connects the Hugging Face Diffusers library with NVIDIA’s NeMo Automodel training library, enabling distributed fine-tuning of diffusion models without checkpoint conversion.

The integration allows users to point to any Diffusers-format model on the Hugging Face Hub and begin training using NeMo Automodel’s PyTorch DTensor-native stack, which supports scaling from a single GPU to hundreds via FSDP2, tensor, expert, context, and pipeline parallelism.

Supported models include FLUX.1-dev and FLUX.2-dev for text-to-image, Wan 2.1 and Wan 2.2 for text-to-video, HunyuanVideo 1.5, and Qwen-Image, with recipes provided for both full fine-tuning and LoRA-style parameter-efficient fine-tuning.

The workflow emphasizes reuse of existing Diffusers YAML configurations and supports round-trip checkpoints that load directly into Diffusers pipelines for inference or back to the Hub for sharing, preserving compatibility with downstream tools like quantization, compilation, LoRA adapters, and custom samplers.

The integration introduces multiresolution bucketed dataloading and latent caching to accelerate throughput, and supports flow-matching as the training objective with latent-space training via pre-encoded VAE outputs.

Users can install NeMo Automodel via a pre-built Docker container, pip, or from source, and follow a documented workflow that begins with dataset pre-encoding, proceeds through training with an existing FLUX YAML config, and ends with generation from the fine-tuned checkpoint.

The collaboration is documented in the Diffusers training guide and is fully open source under the Apache 2.0 license, with contributions from NVIDIA and Hugging Face engineers including Sayak Paul from Hugging Face.

The release highlights practical gains such as no checkpoint conversion, fast path to new model support via small code additions, and scalable training beyond what built-in scripts offer, enabling training of large models like FLUX.1-dev (12B) and HunyuanVideo (13B).