XiangqiBench shows frontier LLMs fail in closed-loop agent evaluations despite strong static metrics

Researchers introduced XiangqiBench, an executable benchmark designed to evaluate LLM agents in closed-loop settings using Chinese chess (xiangqi) endgames with forced mates. Unlike static evaluations that credit a model for naming a correct move, XiangqiBench requires agents to carry a plan through to a verified checkmate while responding to an engine defender.

The benchmark consists of 119 tactical endgames and records 8,568 multi-turn trajectories from 12 frontier LLMs under two observation protocols. An interactive REPL interface separates real moves, state queries, and forward simulation, enabling precise measurement of agent behavior.

Three distinct gaps were identified that overstate agent competence in closed-loop play. First, the Conversion Gap shows that while models play the stored reference first move in 26.1% of Sighted trials, only 13.9% of these trials end in a win, indicating difficulty converting initial correctness into outcomes. Second, the Consistency Gap reveals that the leading model achieves 38.7% pass@3 but only 5.9% pass^3, winning all three trials on just 7 of the 46 positions it ever wins, suggesting instability across repeated attempts. Third, the Simulation Gap highlights that 32.3% of accepted simulation calls stop on an illegal move, and in 49.3% of comparable cases the real defender replies differently from the agent's simulated line, showing that self-authored rollouts cannot anticipate opponent behavior.

The authors argue that static evaluations risk overstating agent capabilities and advocate for closed-loop evaluations that score real outcomes and report reliability alongside coverage, rather than relying on isolated move accuracy or simulated projections.