Allen Institute releases StateBench to test long-context state tracking in language models
StateBench evaluates how models track evolving information in long documents across four difficulty tiers, revealing gaps even in frontier systems.
1 source · cross-referenced
- StateBench introduces a benchmark for long-context state tracking with four difficulty levels.
- Frontier models including Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna struggle on harder state-tracking tasks as inputs grow.
- Recurrent and hybrid architectures like Gated DeltaNet generalize farther than transformers when trained directly on these tasks.
- The benchmark tests open pretrained models and frontier systems on standardized state-tracking problems with sequences of variable assignments.
Allen Institute for AI (AI2) released StateBench, a benchmark designed to evaluate how language models track state changes over long documents. The benchmark measures performance across four difficulty tiers—r-trivial, aperiodic, soluble, and general finite—each representing increasingly complex state update operations such as immutable assignment, reassignment by value, self-referential updates, and assignment by reference.
StateBench was used to test a range of open pretrained models, including transformer and hybrid architectures, as well as frontier systems like Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. Results showed that pretrained models performed well on simpler tiers but struggled on general finite problems, aligning with theoretical predictions about transformer limitations. Hybrid models with Gated DeltaNet layers generalized farther than transformers on these tasks.
The benchmark also evaluated models trained directly on state-tracking tasks. Even in this more favorable setting, accuracy dropped sharply on soluble and general finite problems as input length increased, with performance falling to 13% or lower on the longest general finite problems for frontier models.
StateBench standardizes state-tracking problems by deriving comparable tasks at each difficulty level from the same sequence of variable assignments, ensuring fair comparisons across tiers. The benchmark tests multiple formats for the same underlying problems, such as RULER-style tasks and spreadsheet edits, to account for format sensitivity.
The release includes a technical report, dataset, and code, providing researchers with tools to probe state-tracking capabilities and explore architectural alternatives like recurrent and hybrid designs.
- Nov 6, 2026 · OpenAI — News
OpenAI releases BioBasicsBench to evaluate AI agents on wet-lab science tasks
Trust79 - Nov 3, 2026 · Hugging Face
CWE-bench v2 expands public cybersecurity evaluation suite to 100 tasks across 91 CWEs
Trust79 - Nov 3, 2026 · OpenAI — News
OpenAI releases LifeVal, a benchmark for evaluating AI assistance in everyday tasks
Trust74