Researchers propose RBS-Attention to speed up long-context LLM prefill with radius-bounded sparse attention

Prefill latency in long-context large language model (LLM) inference is increasingly constrained by dense self-attention, which processes the entire prompt before generation begins. Sparse block selection can reduce this cost, but the authors identify a failure mode they term mean dilution: a block centroid may obscure a highly relevant token among many irrelevant ones, leading to underestimation of relevance.

To address this, the authors propose RBS-Attention, a training-free sparse-prefill method that introduces two complementary selection branches. A centroid base branch captures average relevance, while a rescue branch uses the maximum key-block radius and its prompt-, layer-, and head-dependent distribution to identify blocks at risk of underestimation. The method independently thresholds the two branches and combines their masks, preserving regular block-sparse FlashAttention execution.

Evaluated on H100 GPUs, RBS-Attention achieves a 20.65× standalone prefill-attention speedup, an 11.92× speedup over vLLM’s prefill-attention, and a 5.97× end-to-end time-to-first-token speedup at 128K context on the Qwen3-30B-A3B-Instruct-2507-FP8 model. On the dense Qwen3-32B model, RBS-Attention retains 88.65 overall RULER accuracy compared to 89.52 for dense attention, with additional quality evaluations on LongBench-v2, InfiniteBench, and Video-MME.

Supporting experiments characterize retention, compare selectors at matched density, and analyze block-size, threshold, and memory behavior. The authors conclude that radius-adaptive dual-branch selection is an effective approach to long-context prefill.