New benchmark ‘AgentLens’ evaluates interactive coding agents by full task trajectory, not just pass/fail
AgentLens combines formal verification with LLM-written reviews to explain agent behavior across instruction-following, tool use, self-verification, and error recovery.
1 source · cross-referenced
- AgentLens introduces a benchmark that evaluates interactive coding agents by their full task trajectory rather than a binary pass/fail outcome.
- The benchmark pairs formal verification with LLM-generated trajectory reviews and side-by-side comparisons to explain scoring.
- It is designed to diagnose model behavior, compare successive agent versions, and catch product regressions in nightly evaluations.
- The benchmark and code are released open source under the AgentLens project.
A new benchmark called AgentLens evaluates interactive code agents by their full task trajectory—how they follow instructions, use tools, verify their own work, recover from mistakes, and communicate—rather than reducing runs to a binary pass/fail outcome.
The benchmark pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is.
AgentLens is designed to be useful beyond simple model ranking: it is used to diagnose model behavior, compare successive versions of an agent, and catch product regressions in a nightly evaluation pipeline.
The authors release the benchmark as open source at the AgentLens GitHub repository.
- Aug 21, 2026 · Hugging Face
Hugging Face study finds benchmark optimization inflates ASR model scores
Trust84 - Aug 13, 2026 · arXiv cs.CL
Backtrader-Bench introduces self-generating MCQ pipeline to evaluate LLM agents in algorithmic trading
Trust79 - Aug 9, 2026 · Apple — Machine Learning Research
Apple introduces DeepAmbigQA dataset to test LLM answer completeness on ambiguous multi-hop questions
Trust84