Skip to content
Culture · Nov 2, 2026

Measuring AI coding agents by tokens misses the real metric: software that ships

Token-based pricing can distort the value of AI coding agents; tracking cost per merged pull request better reflects ROI and quality.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • Token-based pricing for AI coding agents can obscure true value by conflating consumption with outcomes.
  • A single coding task can consume 10 to 50 times more tokens than a basic chat prompt due to iterative planning, generation, testing, and error correction.
  • Measuring cost per merged pull request—total AI-assisted development cost divided by merged, reviewed changes—better aligns spending with delivered software.
  • Hybrid approaches that route routine work to owned open-weight models and reserve frontier models for complex tasks can reduce token costs by 30 to 50%.
  • Owned capacity becomes more cost-efficient as usage grows, unlike per-token pricing which increases with each call.

When developers assign coding tasks to AI agents, the workflow extends far beyond a single response. Agents plan, generate code, run tests, and iterate to correct errors, often repeating this cycle multiple times. A single task can consume between 10 and 50 times more tokens than a basic chat prompt, according to the article. With per-token pricing models, these high-value workflows can become disproportionately expensive, creating financial pressure that may lead organizations to ration usage through caps, seat limits, or approval gates.

Rationing usage risks restricting the very workflows that deliver the most value, the article argues. Token-based pricing measures consumption, not outcomes, and cannot distinguish between productive iterations and wasteful loops. Instead, the piece proposes measuring cost per merged pull request—total AI-assisted development cost divided by the number of changes that pass review and are merged—as a more accurate gauge of ROI. Tracking this metric alongside rework and defect rates ensures that cost reductions do not come at the expense of quality.

The article highlights Uber as an example of this tension: the company reportedly exhausted its entire 2026 AI budget by April after expanding its use of Claude Code. GitHub Copilot’s shift from flat-rate pricing to metered credits reflects similar industry-wide adjustments to manage rising token costs. Gartner predicts that AI coding token costs will surpass average developer salaries by 2028, underscoring the urgency of rethinking how organizations measure and manage these expenses.

To improve the cost-per-merged-pull-request metric, the article recommends reserving expensive frontier models for tasks requiring advanced planning or novel reasoning, while routing routine work—such as code generation, test creation, and refactoring—to open-weight models operated in-house. This hybrid approach can reduce token costs by 30 to 50%, as demonstrated by HPE’s internal platform, which routes routine workloads from over 150 applications to on-premises open-weight models. A governed gateway can route requests based on task complexity, data sensitivity, cost, and latency, ensuring premium inference is reserved for moments that justify it.

Owned capacity offers additional benefits beyond cost efficiency, the article notes. While per-token pricing makes heavy usage more expensive over time, owned infrastructure spreads costs across usage, becoming more economical as teams scale. Owned capacity also provides control over sensitive code, enabling organizations to keep proprietary code on-premises while only sending governed exceptions to external models. This can unlock adoption in areas where security concerns typically stall progress, such as sensitive repositories.

The article emphasizes that the choice between owned and rented capacity depends on usage patterns. Low or unpredictable usage may favor APIs, while sustained demand can make owned capacity more cost-effective. Most enterprises are expected to adopt a hybrid model: using frontier APIs for demanding or intermittent work and owned capacity for repeatable, high-volume tasks. By aligning metrics with delivered outcomes—such as cost per merged pull request—and optimizing infrastructure accordingly, organizations can scale AI coding agents without falling into the trap of rationing adoption to control costs.

HPE’s experience illustrates the potential of this approach. The company tracks cost per active developer and cost per merged pull request to ensure that AI spending translates into tangible software delivery. Data from HPE shows that the top 5% of users drive most consumption, with the heaviest users generating roughly 450 times more usage than the lightest on the same license. If power users grow from a small minority to half the workforce, monthly spend could rise about 4.4 times with flat headcount—highlighting the need to measure usage patterns and adjust infrastructure accordingly.

Sources
  1. 01MIT Technology Review — AI — Stop counting tokens, start counting what ships
Also on Culture

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.