Researchers propose EvalDetectBench to measure evaluation awareness in frontier language models

Researchers introduced EvalDetectBench, an open pipeline and benchmark to measure evaluation awareness in frontier large language models. Evaluation awareness refers to a model's ability to recognize when it is being evaluated, which can lead to behavior that differs from deployment scenarios. This discrepancy undermines the validity of evaluation results, which are central to current AI safety frameworks.

EvalDetectBench is designed to work with any Inspect-compatible evaluation, enabling practitioners to test against current and future benchmarks. The benchmark includes a newly curated transcript suite that covers current frontier system-card evaluations and diverse deployment sources, providing a standardized way to assess evaluation awareness across different contexts.

The research identifies two methodological choices in existing literature that introduce systematic bias. First, the identity of the model generating deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings. Second, elicitation prompts selected for high performance on one model can perform near chance on others. These biases risk distorting evaluation outcomes and misleading interpretations of model capabilities.

To address these issues, EvalDetectBench incorporates per-model probe calibration and a stratified generator-harmonization procedure. These methods aim to correct for the identified biases, providing more reliable measurements of evaluation awareness and reducing the risk of skewed benchmark results.

The benchmark is released as an open resource, including code and data, to facilitate broader adoption and further research. The authors emphasize the importance of addressing evaluation awareness to ensure that benchmark results accurately reflect model performance in real-world deployment scenarios.