Diffu-LoRA proposes gated low-rank adaptation to personalize diffusion models with fewer parameters

Researchers propose Diffu-LoRA, a parameter-efficient adaptation method for personalizing text-to-image diffusion models using only a few reference images. The approach targets the challenge of preserving subject identity while adhering to new contextual prompts.

The method inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates adaptation weights and gate parameters on separate data splits, and progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen.

In experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets, Diffu-LoRA improved overall subject fidelity and prompt alignment compared to evaluated fine-tuning baselines. Ablation studies examined the contributions of bilevel optimization, progressive pruning, and adapter placement, supporting learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.