Apple proposes ARBITRAGE, a step-level speculative decoding framework to reduce LLM reasoning latency by up to twofold

Apple’s Machine Learning Research group introduced ARBITRAGE, a step-level speculative decoding framework designed to reduce inference latency in large language models (LLMs) during reasoning tasks. The method addresses a limitation of traditional token-level speculative decoding, where semantically equivalent steps are unnecessarily rejected due to token mismatches, leading to wasted target compute.

ARBITRAGE replaces fixed acceptance thresholds with a lightweight router trained to predict when the target model is likely to produce a higher-quality reasoning step. This router approximates an ideal ARBITRAGE ORACLE that always selects the better step, enabling near-optimal efficiency–accuracy trade-offs. The authors report that ARBITRAGE consistently surpasses prior step-level speculative decoding baselines across multiple mathematical reasoning benchmarks.

The framework achieves up to approximately twofold reductions in inference latency at matched accuracy compared to prior methods. The authors note that existing step-level approaches often regenerate many rejected steps with little improvement, which ARBITRAGE mitigates by routing generation dynamically based on relative advantage between draft and target models.

The work is authored by researchers from Apple, UC Berkeley, ICSI, and Lawrence Berkeley National Laboratory, and is presented as a publication dated August 2026. The paper situates ARBITRAGE within the broader context of speculative decoding research, including comparisons to methods such as Mirror Speculative Decoding and Speculative Streaming.