01New benchmark
Turkish decision-making benchmark HakemBench released with 2,346 items across seven tracks
HakemBench is a new Turkish benchmark for evaluating models on typed decision-making tasks, released under CC BY 4.0.
Trust79
HypeLow hype
Archive
01 02 02New benchmark on arXivXiangqiBench shows frontier LLMs fail in closed-loop agent evaluations despite strong static metrics
A new benchmark called XiangqiBench evaluates LLM agents in closed-loop Chinese chess endgames, requiring models to deliver checkmate against an engine defender rather than just suggest a move.
Trust79HypeLow hype03 03Research paper (arXiv cs.CL)Paper proposes Calibrated User Embeddings to improve multi-turn AI agent benchmarking
A new framework called Calibrated User Embeddings (CUE) aims to improve the ecological validity of user simulators used in multi-turn AI agent benchmarks.
Trust79HypeLow hype04 04Third-party reviewReview finds Anthropic’s new auto-eval plugin pushes users to create evals before inspecting data
Anthropic’s claude-api plugin adds `build_eval` and `hill-climb` commands for automating eval design and improvement.
Trust79HypeLow hype05 05New benchmark on arXivScopeBench evaluates agentic security models on scope adherence under goal pressure
ScopeBench introduces 30 agentic security tasks where the objective is only reachable by violating stated scope boundaries.
Trust79HypeLow hype06 06Hugging FaceUK AI Security Institute to publish benchmark results using EvalEval’s open schema
The UK AI Security Institute (AISI) will publish verified evaluation results for five benchmarks and six frontier models using EvalEval’s open reporting schema.
Trust79HypeLow hype07 07arXiv cs.AINew benchmark evaluates language models on interdisciplinary biophysics reasoning
BioPhys-Bridge introduces 500 cases and 1,517 tasks to evaluate evidence-grounded scientific reasoning in biophysics.
Trust79HypeLow hype08 08arXiv cs.AIAnalysis of 14,767 papers finds growing emphasis on action and interaction in LLM benchmarks
A systematic review of 14,767 arXiv papers from January 2022 to August 2026 maps how LLM benchmark design has evolved.
Trust79HypeLow hype09 09Hugging Face BlogBenchMIRT method released to audit what LLM benchmarks actually measure
BenchMIRT is a new method to audit LLM benchmarks at the prompt level using multidimensional Item Response Theory (MIRT).
Trust79HypeLow hype10 10ResearchNew benchmark finds prompt framing and fallacy type strongly influence LLM refusal rates
A new benchmark, DeflectBench, evaluates how large language models respond to prompts requesting rhetorical fallacies like whataboutism, ad hominem, and red herring.
Trust79HypeLow hype11 11AI benchmarksAI models struggle with spatial, memory, and abstract reasoning puzzles designed to expose gaps in capabilities
AI models improved rapidly on New York Times Connections puzzles from 18% solve rate in late 2024 to near-perfect in early 2025.
Trust79HypeLow hype12 12New arXiv paperPaper proposes trajectory-judge framework to expose blind spots in outcome-only LLM agent evaluation
Outcome-only evaluation misses 55% of silent faults in agent trajectories, flagging 33% of correct trajectories as failures.
Trust79HypeLow hype13 13New benchmark on arXivResearchers propose EvalDetectBench to measure evaluation awareness in frontier language models
EvalDetectBench is an open pipeline and benchmark designed to measure evaluation awareness in frontier language models.
Trust79HypeLow hype14 14Google DeepMindGoogle DeepMind pilots first double-blind AI evaluations with cryptographic safeguards
Google DeepMind launched the first double-blind evaluation of a proprietary frontier AI model using cryptographically secure environments.
Trust79HypeLow hype15 15Research announcementNew NL2SQL benchmark shows enterprise database complexity degrades model performance
Introduces ESQ-Bench, an Oracle-first NL2SQL benchmark with three enterprise schema complexity tiers and silent-divergence evaluation.
Trust79HypeLow hype16 16Research paperNew benchmark shows how reader-facing memory formats affect LLM evaluation scores
A new benchmark called RENDER evaluates how the format of memory inputs (raw dialogue, summaries, structured records) affects LLM performance scores.
Trust79HypeLow hype17 17New benchmark on arXivWazobia Eval introduces a 550-example Nigerian Pidgin benchmark for emotion, sarcasm, and cultural reasoning
Wazobia Eval is the first benchmark focused on Nigerian Pidgin emotion understanding, sarcasm detection, and cultural reasoning.
Trust79HypeLow hype18 18Hugging Face researchHugging Face study finds benchmark optimization inflates ASR model scores
Eleven widely used open-source ASR models reproduced benchmark errors and omitted audible words when acoustic cues suggested the test context.
Trust84HypeLow hype19 19Research paperBacktrader-Bench introduces self-generating MCQ pipeline to evaluate LLM agents in algorithmic trading
Backtrader-Bench is a new benchmarking framework for evaluating LLM coding agents in algorithmic trading using self-generated multiple-choice questions (MCQs).
Trust79HypeLow hype20 20Research from Apple MLApple introduces DeepAmbigQA dataset to test LLM answer completeness on ambiguous multi-hop questions
Apple’s ML Research team introduced DeepAmbigQA, a dataset of 3,600 questions designed to test LLMs on ambiguous multi-hop reasoning.
Trust84HypeLow hype21 21New tool releasesmevals framework released for small-scale model, prompt, and harness evaluation
smevals is a new open-source eval suite for testing AI models, prompts, and agent harnesses.
Trust79HypeLow hype22 22Research paperPaper proposes model-consensus framework to rank LLM responses without fixed ground truth
A new arXiv paper introduces a consensus-based evaluation framework that ranks LLM responses by aggregating peer preferences rather than relying on fixed ground truth.
Trust79HypeLow hype23 23Apple ML ResearchApple releases LVSum, a benchmark for timestamp-aware long video summarization with 72 videos across 13 domains
Apple’s Machine Learning Research team introduced LVSum, a human-annotated benchmark for evaluating long-form video summarization with fine-grained temporal alignment.
Trust84HypeLow hype24 24New benchmark on arXivResearchers release CLIR-Bench to evaluate multimodal QA over irregular clinical time series
Introduces CLIR-Bench, a benchmark for multimodal question answering over irregular clinical time series.
Trust79HypeLow hype25 25Hamel HusainComparison finds automated evals correlate with human annotations in 100 traces
An applied AI engineer compared 100 human-annotated traces with automated eval systems to assess their reliability.
Trust79HypeLow hype26 26OpenAI analysisOpenAI flags reliability issues in SWE-Bench Pro coding benchmark
OpenAI published an analysis identifying reliability and accuracy concerns in SWE-Bench Pro, a popular benchmark for evaluating AI coding performance.
Trust79HypeLow hype27 27ResearchNew benchmark ‘AgentLens’ evaluates interactive coding agents by full task trajectory, not just pass/fail
AgentLens introduces a benchmark that evaluates interactive coding agents by their full task trajectory rather than a binary pass/fail outcome.
Trust79HypeLow hype28 28Research announcementNew benchmark CSTutorBench evaluates small language models as tutors for block-based programming
A new benchmark called CSTutorBench evaluates how well small language models can act as tutors in K-12 computer science education using block-based programming in VEX VR.
Trust79HypeLow hype29 29Hugging Face BlogScarfBench released to evaluate AI agents on enterprise Java framework migration
ScarfBench introduces 34 enterprise Java applications, 204 migration tasks, and 1,331 expert-written tests to evaluate AI agents on framework modernization.
Trust79HypeLow hype30 30New benchmark on arXivNew benchmark GPTNT reveals real-time collaboration gaps in multimodal agents
GPTNT is a new benchmark for evaluating real-time collaboration between multimodal agents using the cooperative video game 'Keep Talking and Nobody Explodes'.
Trust79HypeLow hype31 31Research preprintResearchers propose broader evaluation dimensions for AI agents after benchmark accuracy saturates
Accuracy saturation in AI benchmarks often leads to retirement or replacement, but this misses other performance dimensions like construct validity, efficiency, and reliability.
Trust79HypeLow hype32 32Research paperContamination-aware benchmark finds selective abstention in instruction-tuned LLMs
A new benchmark introduces a contamination-aware, multi-zone protocol to evaluate when LLMs should answer or abstain.
Trust79HypeLow hype33 33Apple ML ResearchApple study finds LLM-as-a-judge panels provide roughly two independent votes’ worth of information despite nine judges
A nine-judge LLM-as-a-judge panel provides only about two independent votes’ worth of information due to correlated errors.
Trust84HypeLow hype34 34OpenAIOpenAI releases LifeSciBench, an expert-authored benchmark for evaluating AI in life sciences
OpenAI introduced LifeSciBench, a benchmark designed to evaluate AI systems on real-world life science research tasks and decisions.
Trust75HypeLow hype35 35Hugging Face and IBM ResearchOpen Agent Leaderboard measures full systems, not just models, across diverse real-world tasks
Hugging Face and IBM Research launched the Open Agent Leaderboard, an evaluation framework that benchmarks full agent systems (model plus architecture) rather than isolated models across six real-world task categories.
Trust69HypeLow hype36 36AI EvaluationForum AI recruits top experts to audit foundation models on high-stakes topics like geopolitics and finance
Forum AI, founded by former Meta news chief Campbell Brown 17 months ago, builds benchmarks evaluated by leading experts to test how foundation models perform on high-stakes topics including geopolitics, mental health, finance, and hiring.
Trust53HypeLow hype37 37Benchmark SecurityAutomated red-teaming tool reveals widespread reward-hacking vulnerabilities in AI agent benchmarks
Researchers introduce BenchJack, an automated red-teaming system designed to identify reward-hacking vulnerabilities in AI agent benchmarks across software engineering, web navigation, and other domains.
Trust79HypeLow hype38 38Evaluation BottleneckEvaluation costs, not model training, now dominate AI development budgets
The Holistic Agent Leaderboard spent $40,000 to evaluate 21,730 agent rollouts across 9 models and 9 benchmarks, with single frontier-model runs costing up to $2,829 per task.
Trust70HypeSome hype39 39Applied AI EngineeringEvaluation Framework for LLM Applications: Practical Guidance from 700+ Engineers
Error analysis—systematic manual review of 50-100+ traces—is the foundation of effective LLM evaluation, not infrastructure or metrics.
Trust53HypeLow hype40 40Evaluation MethodsResearchers introduce open-world evaluations to test AI capabilities beyond benchmark saturation
A group of 17 researchers from academia, government, industry, and civil society launched CRUX, a project to conduct 'open-world evaluations' that test AI on complex, unstructured real-world tasks instead of standardized benchmarks.
Trust65HypeLow hype41 41ResearchNew framework quantifies AI agent reliability gaps separate from capability gains
Researchers from Narayanan and Kapoor's lab released a framework decomposing AI agent reliability into 12 dimensions across four core areas: consistency, robustness, calibration, and safety.
Trust65HypeLow hype