Hugging Face and IBM Research launched the Open Agent Leaderboard, an evaluation framework that benchmarks full agent systems (model plus architecture) rather than isolated models across six real-world task categories.
Forum AI, founded by former Meta news chief Campbell Brown 17 months ago, builds benchmarks evaluated by leading experts to test how foundation models perform on high-stakes topics including geopolitics, mental health, finance, and hiring.
Researchers introduce BenchJack, an automated red-teaming system designed to identify reward-hacking vulnerabilities in AI agent benchmarks across software engineering, web navigation, and other domains.
The Holistic Agent Leaderboard spent $40,000 to evaluate 21,730 agent rollouts across 9 models and 9 benchmarks, with single frontier-model runs costing up to $2,829 per task.
Error analysis—systematic manual review of 50-100+ traces—is the foundation of effective LLM evaluation, not infrastructure or metrics.
A group of 17 researchers from academia, government, industry, and civil society launched CRUX, a project to conduct 'open-world evaluations' that test AI on complex, unstructured real-world tasks instead of standardized benchmarks.
Researchers from Narayanan and Kapoor's lab released a framework decomposing AI agent reliability into 12 dimensions across four core areas: consistency, robustness, calibration, and safety.