Researchers propose InflationAgent to cut agentic LLM costs by routing based on token inflation

Researchers introduce InflationAgent, a four-stage router designed to account for "token inflation"—the gap between a model’s per-token price and the true cost of an agentic workflow that retries failed attempts. They define inflation as the ratio of true workflow cost to single-call cost and show that existing systems like FrugalGPT can underestimate real costs by more than twofold on difficult tasks.

The work quantifies inflation across model tiers and task types, reporting inflation as high as 4.25× for a 7B model on multi-hop question answering. To predict inflation before execution, the authors propose CoT Branching Entropy (CBE), a difficulty signal computed entirely from local inference that achieves an AUROC of 0.887 for predicting high inflation.

InflationAgent selects models by maximizing a Semantic Exchange Rate (SER) that divides expected accuracy by predicted true cost, and it implements a fresh-escalation policy that discards failed reasoning chains before routing to a stronger model. On GSM8K under a fixed budget, the router attains 94.7% accuracy compared with 91.0% for FrugalGPT while using 31% fewer tokens.

The authors also evaluate the fresh-escalation policy by measuring the impact of forwarding failed reasoning chains to GPT-4o, which can reduce its accuracy by up to 34.8 percentage points, supporting the design choice to reset chains rather than carry forward partial results.