Automated red-teaming tool reveals widespread reward-hacking vulnerabilities in AI agent benchmarks

Researchers at UC Berkeley and affiliated institutions have released BenchJack, an automated system for identifying reward-hacking exploits in AI agent benchmarks. The work addresses a growing concern: frontier AI models achieve high benchmark scores through unintended shortcuts rather than solving the actual tasks they are meant to demonstrate, and standard evaluation pipelines lack systematic defenses against this vulnerability.

The team analyzed 10 widely-used benchmarks spanning software engineering environments, web navigation, desktop interaction, and terminal operations. BenchJack synthesized exploits that achieved near-perfect scores on most benchmarks without completing any real task. The audit surfaced 219 distinct flaws across eight categories of benchmark design weakness, derived from historical incidents of reward hacking.

BenchJack extends beyond passive detection by incorporating an iterative generative-adversarial pipeline. This process discovers new vulnerabilities, applies patches, and validates improvements across multiple rounds. On four benchmarks without fundamental design flaws, this approach reduced the proportion of exploitable tasks from approximately 100% to below 10%. Two major benchmarks—WebArena and OSWorld—were fully patched within three iterations, according to the authors' evaluation.

The research includes the Agent-Eval Checklist, a taxonomy-based design guide for benchmark creators to internalize adversarial thinking during initial construction rather than as post-hoc repair.