New benchmark ‘AgentLens’ evaluates interactive coding agents by full task trajectory, not just pass/fail
AgentLens combines formal verification with LLM-written reviews to explain agent behavior across instruction-following, tool use, self-verification, and error recovery.
1 source · cross-referenced
- AgentLens introduces a benchmark that evaluates interactive coding agents by their full task trajectory rather than a binary pass/fail outcome.
- The benchmark pairs formal verification with LLM-generated trajectory reviews and side-by-side comparisons to explain scoring.
- It is designed to diagnose model behavior, compare successive agent versions, and catch product regressions in nightly evaluations.
- The benchmark and code are released open source under the AgentLens project.
A new benchmark called AgentLens evaluates interactive code agents by their full task trajectory—how they follow instructions, use tools, verify their own work, recover from mistakes, and communicate—rather than reducing runs to a binary pass/fail outcome.
The benchmark pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is.
AgentLens is designed to be useful beyond simple model ranking: it is used to diagnose model behavior, compare successive versions of an agent, and catch product regressions in a nightly evaluation pipeline.
The authors release the benchmark as open source at the AgentLens GitHub repository.
- Oct 5, 2026 · arXiv cs.CL
Turkish decision-making benchmark HakemBench released with 2,346 items across seven tracks
Trust79 - Oct 5, 2026 · arXiv cs.CL
XiangqiBench shows frontier LLMs fail in closed-loop agent evaluations despite strong static metrics
Trust79 - Oct 5, 2026 · arXiv cs.CL
Paper proposes Calibrated User Embeddings to improve multi-turn AI agent benchmarking
Trust79