Researchers propose Woodpecker Distillation to improve strong-model reasoning by learning from weak-model interventions

Researchers from several institutions propose Woodpecker Distillation, a weak-to-strong training framework designed to diagnose and repair localized reasoning bugs in large language models. The core idea is that many failures in reasoning tasks stem not from global incompetence, but from localized bugs in intermediate steps that can be corrected with targeted interventions.

The method works by inserting short corrective patches generated by a weaker probe model after the same strong-model reasoning prefix. These patches can redirect the trajectory toward a correct solution, but the useful signal is not in the intervention text itself. Instead, the framework learns from how the intervention reshapes the model’s future token predictions.

Woodpecker Distillation contrasts successful and unsuccessful weak-model patches at identical reasoning prefixes, constructs a corrective teacher distribution from their induced future token predictions, and distills this signal into the strong model. This approach aims to capture the underlying corrective dynamics rather than imitating the patch content directly.

In experiments on mathematical reasoning benchmarks, the authors report that Woodpecker Distillation consistently improves strong-model performance and outperforms direct imitation baselines. The results suggest that weak-to-strong distillation can extract meaningful corrective signals from weaker models to enhance stronger ones.