IBM Research details challenges and solutions in building model routers for agentic systems

IBM Research, in a Hugging Face blog post, argues that model routing in agentic systems is not a simple classification problem but a systems optimization challenge. The authors report that real-world routing must account for factors beyond model pricing or task difficulty, including caching behavior, infrastructure state, compliance constraints, and workload patterns.

The post highlights three dimensions where naive routing assumptions break down. First, cost is not solely determined by model pricing; caching can dramatically alter effective costs. In tests using a CodeAct agent on the AppWorld Test Challenge, Claude Sonnet 4.6 incurred $79 total ($0.19 per task) while GPT-4.1 cost $155 ($0.37 per task), despite GPT-4.1 having lower token pricing. The discrepancy stemmed from higher cache-read pricing for Sonnet and longer reasoning steps for GPT-4.1, which reduced cache benefits for the latter.

Second, task difficulty is often invisible at routing time. A seemingly simple request like summarizing a contract may involve retrieval, compliance checks, tool use, and multiple refinement rounds, while a technical prompt might be handled efficiently by a smaller specialized model. Even if difficulty could be estimated, routers must balance cost, latency, model specialization, reliability, and enterprise constraints such as compliance, data residency, privacy, and approved model lists.

Third, latency depends on more than model size. Routing overhead, hardware configuration, cache warmth, endpoint load, and routing granularity all influence end-to-end response times. Routing at every step increases flexibility but adds latency and operational complexity.

IBM Research addressed these challenges by treating routing as an optimization problem rather than a classification task. Their router optimizes across cost, quality, and latency simultaneously while remaining lightweight. On the AppWorld Test Challenge with a CodeAct agent, the router achieved a latency-optimized configuration delivering 84% accuracy for $93 and 83 seconds per task, reducing cost by 21% and latency by 9% compared to running Opus alone, with only a 4% accuracy drop. Another configuration pushed costs lower while maintaining similar accuracy. The router added roughly 6 milliseconds and 2 kilobytes of memory per task, avoiding becoming a bottleneck.

The authors emphasize that effective routing is about optimizing the entire system, not just selecting the "best" model for a task. They plan to share technical details in a follow-up post and invite feedback from developers building routing into agentic systems.