New benchmark and training method show real-time LLM agents struggle when environments evolve during inference

Researchers introduce RealtimeBench, a benchmark with 180 tasks spanning six game types, to evaluate LLM agents in environments that continue evolving during inference. This design explicitly tests whether agents can act effectively under real-time constraints, where delayed actions may become obsolete.

In paired evaluations, a stronger model (GPT-6.1-sol) outperforms a faster model (Jev) when environments are paused, but the ranking reverses when environments run live. This suggests that raw capability does not guarantee real-time effectiveness when reasoning and tool use introduce latency.

To address this, the authors propose Two-Timescale Offline Recursive Self-Improvement (TORSI), a method that separates deliberation from action. A slow teacher analyzes failures and tests corrections in paused environments between attempts, then writes reusable instructions for a fast student.

The student agent executes actions without teacher calls during live gameplay, and validation scores determine whether updates are adopted. After two rounds of TORSI, Jev’s average test score increases from 13.51 to 24.19 on a 0–100 progress scale, with improvements observed across all six game types.

The same set of instructions derived from TORSI also improves average scores for three other models without additional tuning, indicating that the approach generalizes beyond the base agent.