BenchMIRT method released to audit what LLM benchmarks actually measure

Allen Institute for AI (AI2) released BenchMIRT, a method that audits LLM benchmarks at the level of individual prompts using multidimensional Item Response Theory (MIRT).

The authors note that benchmarks are often designed to measure a stated ability—such as safety or general reasoning—but individual tasks within them may depend on additional, unintended capabilities. For example, BBQ includes questions that probe age bias but also require tracking entities and reasoning from evidence, which may reflect general reasoning more than safety.

BenchMIRT extends prior work that applied single-dimensional IRT to benchmarks by modeling multiple underlying capabilities simultaneously. It estimates both model-level abilities and question-level difficulty and discrimination across two dominant dimensions—safety and general reasoning—without being told which benchmarks correspond to which capability.

The team trained BenchMIRT on benchmarking results from 100 open-weight LLMs across 16 benchmarks and more than 34,000 questions. Six of the benchmarks measured general reasoning (e.g., MMLU-Pro, GPQA, MATH, BBH), and 10 came from AI2’s safety suite (e.g., HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, XSTest).

The analysis recovered two stable dimensions—safety and general reasoning—that emerged consistently across repeated runs, indicating the method’s robustness.

BenchMIRT’s findings challenge some assumptions about benchmark alignment. For instance, BBQ, commonly grouped with safety benchmarks, aligned more strongly with general reasoning, suggesting low BBQ scores may partly reflect reasoning difficulty rather than safety behavior alone.

Similarly, WMDP, which tests dangerous dual-use knowledge, aligned more with general reasoning than safety in the analysis. The authors note that stronger general reasoning was associated with lower WMDP scores because the benchmark counts refusal or failure to provide the dangerous knowledge as the desired response.

HarmBench includes different question types: standard harmful requests, contextual harmful requests, and copyright-related requests. BenchMIRT found that both standard and contextual HarmBench questions aligned more closely with safety, while copyright-related questions aligned more closely with general reasoning.

Beyond auditing, BenchMIRT can identify the most informative questions in a benchmark. By retaining only 10% of questions across the 16 benchmarks, the method preserved nearly the same model rankings on the underlying safety or reasoning capabilities as using the full set. Retaining 50% of questions often matched the full benchmark’s measures even more closely.

The authors also report that BenchMIRT correctly predicted whether a model would answer a held-out benchmark question correctly 79% of the time, outperforming a baseline that assumed uniform performance across questions (70% accuracy).