New Realtime-Bench shows frontier AI models struggle with low-latency tasks
Researchers introduce a benchmark measuring real-time decision-making in interactive settings, finding even top models succeed less than 40% of the time under realistic latency.
1 source · cross-referenced
- A new benchmark called Realtime-Bench evaluates how well AI models perform tasks requiring real-time decisions, such as online auctions or games.
- Top models achieved only a 36% success rate under realistic latency conditions, compared to up to 80% when time constraints are removed.
- The benchmark measures three qualities: temporal state awareness, adaptive deliberation, and time-aware action selection.
- Researchers from UT Austin, Ohio State, UIUC, NVIDIA, Northwestern, and Steadyworks developed the benchmark.
- The findings suggest AI agents are not yet reliable for time-sensitive applications.
Researchers from the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks introduced Realtime-Bench, a benchmark designed to evaluate AI models on tasks requiring real-time decision-making.
The benchmark consists of 50 interactive tasks, including 30 web-based tasks across travel, e-commerce, service, and security, and 20 game-based tasks spanning arcade, puzzle, strategy, and simulation genres.
Realtime-Bench is structured to test three core capabilities: temporal state awareness (RT-Perceive), adaptive deliberation (RT-Deliberate), and time-aware action selection (RT-Act). These capabilities are essential for tasks where delayed actions directly impact success, such as purchasing a ticket while tracking its sale status or deciding when to act during an online auction.
The benchmark evaluates models under three conditions: wall clock (real-world latency), paused clock (baseline capability without time pressure), and token clock (simulated time advancing per generation token, regardless of actual inference latency).
When tested under wall clock conditions, the highest-performing model, GPT-6 Astra (high), achieved a 36% pass rate. In contrast, under paused clock conditions, models with high reasoning capabilities approached success rates of 70% to 80%, indicating a significant performance gap under realistic latency constraints.
The benchmark tested a range of models, including OpenAI’s GPT-6 Astra, GPT-5.6 Sol, GPT-Realtime-2, and OpenAI Decisions; Anthropic’s Claude Fable 5.1 and Claude Opus 5; Google’s Gemini 3.8 Flash, Gemini 3.1 Flash Live, and Gemini 3.8 Live; Jev 1.13; and Chinese models from Deepseek (V4.1) and Moonshot AI (Kimi-K3).
The researchers note that the benchmark serves as a proxy for identifying where AI agents are likely to fail, emphasizing that current systems are not yet reliable for time-sensitive applications.
- Nov 11, 2026 · Wired
EU proposes Digital Fairness Act to ban dark patterns and restrict addictive design features
Trust75 - Nov 11, 2026 · AI Snake Oil — Narayanan & Kapoor
New nonprofit STABL launches to study societal bottlenecks of advanced AI
Trust79 - Nov 10, 2026 · TechCrunch — AI
New York enforces AI safety law requiring registration and incident reporting for frontier labs
Trust75