VarRate proposes training-free variable-rate KV cache compression for long-context LLMs

The key-value (KV) cache is a primary memory bottleneck during long-context LLM inference, and existing training-free compression families face structural limitations. Token-selection methods such as SnapKV and Ada-KV score token importance from an observation window and irreversibly evict low-scoring tokens, which leads to accuracy collapses of 11–15 points when importance signals degrade under query-agnostic reuse. Uniform low-rank coding avoids eviction but allocates equal rank to every token, wasting budget.

VarRate proposes a training-free KV codec that assigns each token a variable low-rank budget based on query salience while keeping every token at a nonzero rank. Unlike comparable adaptive-rank codecs that require training to achieve such allocation, VarRate achieves this without any training. Because no token is dropped, its accuracy degradation is limited to 3.5–5.5 points in scenarios where token-selection methods collapse by 11–15 points.

On LongBench across 16 tasks with a matched 20% budget, VarRate remains within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two models, VarRate is described as the strongest matched-memory compressor in the paper. It also significantly outperforms its uniform-rank ablation on both models.

Compared to KVzip, a method designed for query-agnostic reuse, VarRate is accuracy-equivalent in three of four settings and within one point overall, while incurring about one-eighth the prefill overhead. The paper includes 20 pages, 8 figures, and 24 tables, with additional experiments and analyses provided in an appendix.