Hugging Face-backed team proposes memory-efficient knowledge distillation for large language models

A Hugging Face–affiliated team proposes two systems-level changes to make knowledge distillation of large language models far more memory-efficient. The first change is offline distillation: compute the teacher’s output once, cache only the top-100 most likely tokens per position, and train the student against that cache, eliminating the need to keep the teacher in memory during training. The second change is a fused chunked KL loss that processes the sequence in chunks, projects hidden states to logits for one chunk at a time, folds the result into the running loss, and discards the chunk before moving to the next. The backward pass recomputes each chunk on the fly, so peak memory grows only linearly with sequence length instead of spiking with the full vocabulary × sequence size.

The authors quantify the gains on a single H200 GPU using Llama 3.1 8B Instruct as the teacher and a 3.2B Llama model as the student at an 8K token context. Compared with online distillation, the fused chunked KL loss reduces peak VRAM from roughly 250GB to about 128GB while preserving training loss fidelity. Across four setups—online distillation and three offline variants (dense KL, forward-chunked KL, fused chunked KL)—the loss curves overlap almost exactly, indicating that offline distillation with top-100 cached logits is lossless relative to online distillation.

The team reports iteration times and throughput for each method. Offline, fused chunked KL reaches about 304 TFLOP/s with 20.2 seconds per iteration, compared to 237 TFLOP/s and 25.9 seconds for online distillation. Peak memory for the fused approach is 58.3 GB versus 102.8 GB for online distillation in this configuration.

To support adoption, the authors have open-sourced the fused chunked KL loss implementation on GitHub under CompactifAI/Full-Chunked-KL-Loss.