Apple researchers propose training-free diagnostic framework to analyze on-policy distillation for reasoning models

Apple Machine Learning Research published a paper introducing a training-free diagnostic framework to analyze on-policy distillation in reasoning models. The framework evaluates distillation signals at the highest resolution: per token, per question, and per teacher. The authors derive an ideal per-node gradient defined as the parameter update that maximally increases the student’s probability of success. They then propose a scalable targeted-rollout algorithm to estimate this gradient efficiently, even for long chains of intermediate thoughts.

The paper introduces a gradient alignment score, defined as the cosine similarity between the ideal gradient and any given distillation gradient. This score quantifies how closely a particular distillation configuration approximates the ideal signal. Across experiments in self-distillation and external teacher models, the authors observe that distillation guidance exhibits substantially higher alignment with the ideal on incorrect rollouts than on correct ones. They attribute this to the student already performing well in correct rollouts, where the teacher’s signal tends to become noisy.

The findings further indicate that the optimal distillation context depends jointly on the student model’s capacity and the target task. No single universally effective configuration emerged across settings, motivating the use of per-task, per-token diagnostic analyses for distillation. The authors suggest that these insights can guide practitioners in selecting distillation strategies without resorting to costly training runs.

The paper is authored by Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N.M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, and Mehrdad Farajtabar, with equal contribution noted among the first two authors. It was published in July 2026 and is available on the Apple Machine Learning Research publications page.