Analysis of 14,767 papers finds growing emphasis on action and interaction in LLM benchmarks

A new arXiv paper systematically maps 14,767 papers introducing or updating LLM evaluation resources submitted between January 2022 and August 2026, offering a quantitative view of how benchmark design has shifted alongside model capabilities. The analysis uses staged screening and automated full-text coding to categorize changes in target systems, domains, evaluation materials, conditions, and scoring mechanisms.

The study finds a growing emphasis on action, interaction, and professional applications in benchmark design, reflecting researchers’ evolving expectations for what LLMs should be able to do. Established and newer design elements are observed to coexist, indicating a layered and uneven transition in evaluation practices rather than a clean break from prior methods.

Model participation in evaluation is developing unevenly: the use of LLM-based scoring is increasing within both agent and non-agent benchmark groups, suggesting a broader reliance on AI systems to judge performance. By contrast, the use of model-generated materials in benchmarks has not seen a comparable sustained increase in recent cohorts, indicating that human-curated content remains dominant despite the rise of AI-assisted evaluation.

The authors argue that as AI systems take on roles in constructing tests, performing tasks, and judging responses, the expansion of evaluation practices may not necessarily yield more independent evidence of capability. Instead, it could risk reproducing the preferences and blind spots of the very models being evaluated, raising methodological and epistemological questions about the future of LLM benchmarking.