Researchers propose a closed-loop framework to link evaluation failures to targeted data interventions in LLM training

Model capability in large language models (LLMs) is shaped prospectively by training data but observed retrospectively through evaluation, which compresses complex factors like prompts, decoding, and scoring into a single noisy score. This disconnect forces engineers to infer data fixes from failures using intuition rather than method. To close this gap, the authors introduce the concept of a *capability slice*: a group of evaluation samples that share a background condition, task type, solving operation, and output constraint. This unit is designed to be precise enough to localize a single weakness while remaining stable enough to aggregate meaningfully, unlike coarse benchmark names or noisy single-sample evaluations.

The proposed framework consists of an evaluation taxonomy, a non-instruction data taxonomy, and mapping rules that together form a closed loop. This loop converts a benchmark-level failure into a targeted, testable intervention—either in the data or the training process. The authors validate the approach through two case studies that test the loop in opposite directions. In the first case, continued pre-training caused a 46.82% drop in BBH scores, but diagnosis traced the issue to a single masked end-of-sequence token loss rather than degraded reasoning. Restoring this loss recovered BBH to 66.44, surpassing the original checkpoint without altering the training data.

In the second case, the loop identified a persistent math-reasoning weakness and decomposed it by solving operation into specific failing combinations. A weakness-targeted sampling procedure built from this analysis improved AIME2025 and AIME2026 Pass@128 scores from 6.67 and 0.00 to 26.67 each. The same unmodified loop produced correct, opposite verdicts in both scenarios, demonstrating that evaluation-to-data inference can be made routine, auditable, and experimentally validated rather than reliant on intuition.

The authors argue that this closed-loop methodology reduces the reliance on post-hoc intuition in LLM development and provides a framework for systematically addressing model weaknesses with measurable outcomes.