Skip to content
Research · Oct 28, 2026

Paper proposes ledger-based framework to measure time saved by AI assistance

New arXiv pre-print argues for a counterfactual 'ledger view' of time savings, contrasting it with stopwatch metrics and offering model-free quantities to quantify trade-offs between speed and error.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • A new arXiv pre-print introduces a 'ledger view' to measure time saved by AI assistance, arguing that time savings must account for errors introduced.
  • The paper defines model-free quantities—time saved and excess escaped defects—and a decision metric, the break-even defect cost.
  • Reanalysis of METR’s developer study shows AI assistance could slow work by 19% to 24% depending on the estimand, with help rates ranging from 3% to 82%.
  • In GDPval’s review workflow, review time alone caps GPT-5’s speedup at 3.6x; faster but less accurate consultants were observed.
  • A small pre-specified experiment found reviewers reduced effort when told error rates were lower, leading to more errors slipping through.

A new arXiv pre-print proposes a counterfactual framework for measuring time saved by AI assistance, arguing that traditional 'stopwatch' metrics ignore the errors introduced by AI systems. The author introduces a 'ledger view,' which treats time savings as a counterfactual that must be evaluated alongside the errors the system allows to escape. This approach defines two model-free quantities: time saved and excess escaped defects, and derives a decision metric called the break-even defect cost to guide whether AI assistance is net beneficial.

The paper critiques the use of randomized studies for identifying time savings, showing what such studies can and cannot measure. It reanalyzes METR’s developer study using public data, finding that the same 246 issues support estimates ranging from a 19% slowdown to a 24% slowdown depending on the estimand, and that AI assistance helped on between 3% and 82% of issues. Without adjusting for a chance imbalance, the slowdown estimate rises to 34%.

The author applies the framework to GDPval’s review-every-output workflow, showing that review time alone caps GPT-5’s potential speedup at 3.6x. Consultants outside the frontier were observed to be faster but more often wrong, illustrating the trade-off between speed and accuracy.

The paper tests two predictions in a small pre-specified experiment where AI reviewers checked AI-generated drafts derived from SEC filings made after the models’ training data cutoff. When reviewers were told the drafter’s true error rate, they checked more as the error rate rose, but did so in an all-or-nothing manner, switching near the break-even point of the prompt’s costs. When told the drafter erred ten times less often than it actually did, reviewers checked 7 fewer of 9 figures and allowed a wrong figure through in 22.5% of briefs instead of none, demonstrating faster but worse outcomes driven by reduced effort.

Sources
  1. 01arXiv cs.AI — Counting the Bandages: On measuring the time that AI saves
Also on Research

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.