01New benchmark
Turkish decision-making benchmark HakemBench released with 2,346 items across seven tracks
HakemBench is a new Turkish benchmark for evaluating models on typed decision-making tasks, released under CC BY 4.0.
Trust79
HypeLow hype
Archive
01 02 02New benchmark on arXivXiangqiBench shows frontier LLMs fail in closed-loop agent evaluations despite strong static metrics
A new benchmark called XiangqiBench evaluates LLM agents in closed-loop Chinese chess endgames, requiring models to deliver checkmate against an engine defender rather than just suggest a move.
Trust79HypeLow hype03 03Research paper (arXiv cs.CL)Paper proposes Calibrated User Embeddings to improve multi-turn AI agent benchmarking
A new framework called Calibrated User Embeddings (CUE) aims to improve the ecological validity of user simulators used in multi-turn AI agent benchmarks.
Trust79HypeLow hype04 04Third-party reviewReview finds Anthropic’s new auto-eval plugin pushes users to create evals before inspecting data
Anthropic’s claude-api plugin adds `build_eval` and `hill-climb` commands for automating eval design and improvement.
Trust79HypeLow hype05 05New benchmark on arXivScopeBench evaluates agentic security models on scope adherence under goal pressure
ScopeBench introduces 30 agentic security tasks where the objective is only reachable by violating stated scope boundaries.
Trust79HypeLow hype06 06Hugging FaceUK AI Security Institute to publish benchmark results using EvalEval’s open schema
The UK AI Security Institute (AISI) will publish verified evaluation results for five benchmarks and six frontier models using EvalEval’s open reporting schema.
Trust79HypeLow hype07 07arXiv cs.AINew benchmark evaluates language models on interdisciplinary biophysics reasoning
BioPhys-Bridge introduces 500 cases and 1,517 tasks to evaluate evidence-grounded scientific reasoning in biophysics.
Trust79HypeLow hype08 08arXiv cs.AIAnalysis of 14,767 papers finds growing emphasis on action and interaction in LLM benchmarks
A systematic review of 14,767 arXiv papers from January 2022 to August 2026 maps how LLM benchmark design has evolved.
Trust79HypeLow hype09 09Hugging Face BlogBenchMIRT method released to audit what LLM benchmarks actually measure
BenchMIRT is a new method to audit LLM benchmarks at the prompt level using multidimensional Item Response Theory (MIRT).
Trust79HypeLow hype10 10ResearchNew benchmark finds prompt framing and fallacy type strongly influence LLM refusal rates
A new benchmark, DeflectBench, evaluates how large language models respond to prompts requesting rhetorical fallacies like whataboutism, ad hominem, and red herring.
Trust79HypeLow hype