Quantization-aware healing yields a 4-bit LLM that outperforms its full-precision original on most benchmarks

MultiverseComputingCAI introduces Quantization-Aware Healing (QAH), a method that compresses a GPT-OSS 120B model to 60B parameters, quantizes it to 4-bit MXFP4, and then heals it by distilling directly from the original full-precision teacher rather than from the intermediate recovered checkpoint.

The authors argue that standard healing pipelines—compress, quantize, then heal—degrade capabilities in reasoning, math, and code generation, and that existing healing techniques like quantization-aware training (QAT) or quantization-aware distillation (QAD) hit ceilings when structural compression precedes quantization.

QAT re-runs expensive post-training stages through a noisy, lower-precision forward pass and can become unstable with prolonged training, while QAD relies on a full-precision teacher of the exact same architecture, which does not exist after structural compression.

QAH sidesteps these issues by distilling the student directly from the original full-size, full-precision model, even though teacher and student differ in size and precision. The student learns only from the teacher’s output distribution via KL divergence on logits, avoiding hard labels and stabilizing training once the student catches up to the teacher.

To handle long contexts up to 32k tokens, the authors reuse a memory-efficient, chunked KL-divergence loss from prior work, which computes KL over sequence slices without materializing the full vocabulary-by-sequence grid.

In head-to-head results on nine benchmarks, the 4-bit QAH model beats its own 16-bit recovery checkpoint on seven tasks, including +7.4 points on AA-LCR (long-context reasoning) and +5.6 points on AIME 2025 (math). The two losses are less than 1.5 points on MMLU-Pro and SciCode.

Despite running at half the parameter count and roughly a quarter the weight memory of the original 120B teacher, the QAH model surpasses the teacher on LiveCodeBench (66.5 vs. 66.0) and trails by only 1.6 points on GPQA Diamond (67.4 vs. 69.0).