Paper proposes behavioral stress tests for forecasting agents, finds mechanism choice is source-dependent

A new paper on arXiv proposes behavioral stress tests for forecasting agents, arguing that the choice to retrieve, reason, defer to a market prior, or use a historical analog should be treated as an observable behavior rather than a hidden implementation detail.

The study, titled 'When Should Forecasting Agents Reason? Behavioral Stress Tests for Reliability Routing,' evaluates mechanism choice across ForecastBench-style binary forecasting tasks and finds that the optimal mechanism is source-dependent. Structured analogs perform best for some data-generating processes, while market/crowd-style and conservative baselines are more effective for others.

The authors introduce ReliabilityRoute, a structural intervention designed to steer forecasting-agent behavior using reliability features such as historical coverage, market-prior availability, source-prior sharpness, evidence strength, evidence disagreement, and horizon.

Two routing policies are evaluated: a fixed rule fitted to 2024 data, which closely matches a hand taxonomy without relying on hard-coded source-name decisions, and a walk-forward self-adjusting rule that refits thresholds from previously resolved vintages. The latter achieves the best mean Brier score among deterministic systems across 16 later LLM vintages.

The reported gains are modest, and historical/search baselines remain highly competitive, suggesting that the primary contribution lies in the behavioral stress test framework rather than performance breakthroughs.

The paper emphasizes that more reasoning is not always better and advocates for forecasting agents to first estimate which evidence source deserves control. It also argues that routing policies should adapt under auditable constraints, with reproducibility artifacts available at a provided GitHub repository.