Skip to content
Policy · Nov 10, 2026

New Realtime-Bench shows frontier AI models struggle with low-latency tasks

Researchers introduce a benchmark measuring real-time decision-making in interactive settings, finding even top models succeed less than 40% of the time under realistic latency.

Trust75
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • A new benchmark called Realtime-Bench evaluates how well AI models perform tasks requiring real-time decisions, such as online auctions or games.
  • Top models achieved only a 36% success rate under realistic latency conditions, compared to up to 80% when time constraints are removed.
  • The benchmark measures three qualities: temporal state awareness, adaptive deliberation, and time-aware action selection.
  • Researchers from UT Austin, Ohio State, UIUC, NVIDIA, Northwestern, and Steadyworks developed the benchmark.
  • The findings suggest AI agents are not yet reliable for time-sensitive applications.

Researchers from the University of Texas at Austin, Ohio State University, University of Illinois at Urbana-Champaign, NVIDIA, Northwestern University, and Steadyworks introduced Realtime-Bench, a benchmark designed to evaluate AI models on tasks requiring real-time decision-making.

The benchmark consists of 50 interactive tasks, including 30 web-based tasks across travel, e-commerce, service, and security, and 20 game-based tasks spanning arcade, puzzle, strategy, and simulation genres.

Realtime-Bench is structured to test three core capabilities: temporal state awareness (RT-Perceive), adaptive deliberation (RT-Deliberate), and time-aware action selection (RT-Act). These capabilities are essential for tasks where delayed actions directly impact success, such as purchasing a ticket while tracking its sale status or deciding when to act during an online auction.

The benchmark evaluates models under three conditions: wall clock (real-world latency), paused clock (baseline capability without time pressure), and token clock (simulated time advancing per generation token, regardless of actual inference latency).

When tested under wall clock conditions, the highest-performing model, GPT-6 Astra (high), achieved a 36% pass rate. In contrast, under paused clock conditions, models with high reasoning capabilities approached success rates of 70% to 80%, indicating a significant performance gap under realistic latency constraints.

The benchmark tested a range of models, including OpenAI’s GPT-6 Astra, GPT-5.6 Sol, GPT-Realtime-2, and OpenAI Decisions; Anthropic’s Claude Fable 5.1 and Claude Opus 5; Google’s Gemini 3.8 Flash, Gemini 3.1 Flash Live, and Gemini 3.8 Live; Jev 1.13; and Chinese models from Deepseek (V4.1) and Moonshot AI (Kimi-K3).

The researchers note that the benchmark serves as a proxy for identifying where AI agents are likely to fail, emphasizing that current systems are not yet reliable for time-sensitive applications.

Sources
  1. 01Import AI — Jack Clark — Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4
Also on Policy

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.