Apple’s Machine Learning Research team introduced LVSum, a human-annotated benchmark for evaluating long-form video summarization with fine-grained temporal alignment.
Introduces CLIR-Bench, a benchmark for multimodal question answering over irregular clinical time series.
An applied AI engineer compared 100 human-annotated traces with automated eval systems to assess their reliability.
OpenAI published an analysis identifying reliability and accuracy concerns in SWE-Bench Pro, a popular benchmark for evaluating AI coding performance.
AgentLens introduces a benchmark that evaluates interactive coding agents by their full task trajectory rather than a binary pass/fail outcome.
A new benchmark called CSTutorBench evaluates how well small language models can act as tutors in K-12 computer science education using block-based programming in VEX VR.
ScarfBench introduces 34 enterprise Java applications, 204 migration tasks, and 1,331 expert-written tests to evaluate AI agents on framework modernization.
GPTNT is a new benchmark for evaluating real-time collaboration between multimodal agents using the cooperative video game 'Keep Talking and Nobody Explodes'.
Accuracy saturation in AI benchmarks often leads to retirement or replacement, but this misses other performance dimensions like construct validity, efficiency, and reliability.
A new benchmark introduces a contamination-aware, multi-zone protocol to evaluate when LLMs should answer or abstain.
A nine-judge LLM-as-a-judge panel provides only about two independent votes’ worth of information due to correlated errors.
OpenAI introduced LifeSciBench, a benchmark designed to evaluate AI systems on real-world life science research tasks and decisions.
Hugging Face and IBM Research launched the Open Agent Leaderboard, an evaluation framework that benchmarks full agent systems (model plus architecture) rather than isolated models across six real-world task categories.
Forum AI, founded by former Meta news chief Campbell Brown 17 months ago, builds benchmarks evaluated by leading experts to test how foundation models perform on high-stakes topics including geopolitics, mental health, finance, and hiring.
Researchers introduce BenchJack, an automated red-teaming system designed to identify reward-hacking vulnerabilities in AI agent benchmarks across software engineering, web navigation, and other domains.
The Holistic Agent Leaderboard spent $40,000 to evaluate 21,730 agent rollouts across 9 models and 9 benchmarks, with single frontier-model runs costing up to $2,829 per task.
Error analysis—systematic manual review of 50-100+ traces—is the foundation of effective LLM evaluation, not infrastructure or metrics.
A group of 17 researchers from academia, government, industry, and civil society launched CRUX, a project to conduct 'open-world evaluations' that test AI on complex, unstructured real-world tasks instead of standardized benchmarks.
Researchers from Narayanan and Kapoor's lab released a framework decomposing AI agent reliability into 12 dimensions across four core areas: consistency, robustness, calibration, and safety.