Paper proposes ledger-based framework to measure time saved by AI assistance
New arXiv pre-print argues for a counterfactual 'ledger view' of time savings, contrasting it with stopwatch metrics and offering model-free quantities to quantify trade-offs between speed and error.
1 source · cross-referenced
- A new arXiv pre-print introduces a 'ledger view' to measure time saved by AI assistance, arguing that time savings must account for errors introduced.
- The paper defines model-free quantities—time saved and excess escaped defects—and a decision metric, the break-even defect cost.
- Reanalysis of METR’s developer study shows AI assistance could slow work by 19% to 24% depending on the estimand, with help rates ranging from 3% to 82%.
- In GDPval’s review workflow, review time alone caps GPT-5’s speedup at 3.6x; faster but less accurate consultants were observed.
- A small pre-specified experiment found reviewers reduced effort when told error rates were lower, leading to more errors slipping through.
A new arXiv pre-print proposes a counterfactual framework for measuring time saved by AI assistance, arguing that traditional 'stopwatch' metrics ignore the errors introduced by AI systems. The author introduces a 'ledger view,' which treats time savings as a counterfactual that must be evaluated alongside the errors the system allows to escape. This approach defines two model-free quantities: time saved and excess escaped defects, and derives a decision metric called the break-even defect cost to guide whether AI assistance is net beneficial.
The paper critiques the use of randomized studies for identifying time savings, showing what such studies can and cannot measure. It reanalyzes METR’s developer study using public data, finding that the same 246 issues support estimates ranging from a 19% slowdown to a 24% slowdown depending on the estimand, and that AI assistance helped on between 3% and 82% of issues. Without adjusting for a chance imbalance, the slowdown estimate rises to 34%.
The author applies the framework to GDPval’s review-every-output workflow, showing that review time alone caps GPT-5’s potential speedup at 3.6x. Consultants outside the frontier were observed to be faster but more often wrong, illustrating the trade-off between speed and accuracy.
The paper tests two predictions in a small pre-specified experiment where AI reviewers checked AI-generated drafts derived from SEC filings made after the models’ training data cutoff. When reviewers were told the drafter’s true error rate, they checked more as the error rate rose, but did so in an all-or-nothing manner, switching near the break-even point of the prompt’s costs. When told the drafter erred ten times less often than it actually did, reviewers checked 7 fewer of 9 figures and allowed a wrong figure through in 22.5% of briefs instead of none, demonstrating faster but worse outcomes driven by reduced effort.
- Nov 11, 2026 · The Verge — AI
Google DeepMind unveils AlphaGenome 2 to predict genetic variant effects on cell types
Trust79 - Nov 11, 2026 · Google DeepMind — Blog
Google DeepMind unveils AlphaGenome 2 to predict DNA variant effects in specific cell types
Trust84 - Nov 11, 2026 · arXiv cs.AI
Researchers demonstrate auditable human-agent teams for autonomous synchrotron X-ray experiments
Trust79