New benchmark ‘AgentLens’ evaluates interactive coding agents by full task trajectory, not just pass/fail

A new benchmark called AgentLens evaluates interactive code agents by their full task trajectory—how they follow instructions, use tools, verify their own work, recover from mistakes, and communicate—rather than reducing runs to a binary pass/fail outcome.

The benchmark pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is.

AgentLens is designed to be useful beyond simple model ranking: it is used to diagnose model behavior, compare successive versions of an agent, and catch product regressions in a nightly evaluation pipeline.

The authors release the benchmark as open source at the AgentLens GitHub repository.