Researchers propose TreeGraft to accelerate LLM inference with adaptive multi-drafter tree-based speculative decoding

Speculative decoding accelerates large language model (LLM) inference using a draft-then-verify paradigm, but tree-structured methods that organize proposals into multiple candidate paths face a trade-off: smaller drafters are fast but produce lower-quality draft trees, while larger drafters improve tree quality at the cost of higher latency.

Existing tree-based speculative decoding methods rely on a single drafter for all drafting steps, which limits flexibility and efficiency. TreeGraft, proposed by researchers from multiple institutions, introduces a multi-drafter framework where drafters of varying computational costs collaboratively construct a shared draft tree.

The framework uses a stronger drafter to rescore candidates, reselect grafting positions, and recover promising paths that may have been overlooked by weaker drafters. It also integrates stronger drafter expansions without disrupting existing branches that could still be accepted by the target model, preserving potential high-quality paths.

To manage computational overhead, TreeGraft employs a lightweight scheduler distilled from an offline value system to determine when to invoke the stronger drafter, balancing quality and efficiency.

Evaluated across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of two fixed single-drafter strategies by an average of 15.1%, with a maximum gain of 26.6%. The authors provide open-source code for further experimentation and adoption.