Researchers propose using model transcripts to evaluate capabilities after benchmark saturation

A new paper on arXiv proposes a method to extract capability-relevant signals from model transcripts even after benchmark scores have saturated, addressing a growing challenge in AI evaluation. The authors argue that traditional benchmark accuracy scores lose their discriminative power as models approach ceiling performance, reducing their utility for differentiating capabilities and informing safety cases or frontier policies.

To test their hypothesis, the researchers simulate benchmark saturation retrospectively using historical benchmark and model releases. They then evaluate protocols that analyze only the transcripts of correctly solved tasks—pairwise judging, single-transcript absolute scoring, and predictions from deterministic transcript features or embeddings—against held-out, unsaturated benchmark items.

Among the evaluated protocols, pairwise judging performed best, reducing prediction error by 5.3 times compared to a baseline that relies solely on saturated benchmark scores. The authors attribute this improvement to the method’s ability to compare model behaviors directly, rather than relying on aggregate accuracy metrics.

The study also finds that capability-relevant information is distributed across transcript sections but concentrated in specific channels such as reasoning and tool-calls. However, the predictive power of these protocols does not always generalize across distribution shifts—for example, underestimating the capabilities of early 2025 reasoning models.

The paper concludes that while saturated benchmarks retain useful signal in their task transcripts, extending their lifetime requires methods explicitly designed to extract these signals robustly. The authors emphasize the need for further research to improve generalization across evolving model capabilities and evaluation scenarios.