Researchers propose Kara, a sliding-window KV cache compression method to improve reasoning LLM serving efficiency

Long chain-of-thought (CoT) generation in reasoning language models leads to the accumulation of large KV caches during decoding, which increases memory overhead, decoding latency, and limits throughput. Existing KV cache compression techniques attempt to mitigate this by selectively removing unimportant KV pairs, but often introduce new challenges.

The authors identify two key limitations in prior methods: threshold-triggered compression policies may yield limited throughput gains or even reduce throughput, and they frequently eliminate entire blocks of KV pairs, exacerbating information loss. Additionally, these methods typically preserve either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to retain important semantic information at arbitrary token positions.

To overcome these limitations, the researchers propose Kara, a sliding-window KV cache compression method that operates on recently generated context during decoding. Kara uses bidirectional attention to score and select informative KV pairs within the window, enabling more precise retention of relevant information.

The authors further design a Token2Chunk module to expand selected KV pairs into flexible-sized chunks, preserving important semantic content more effectively than rigid chunking strategies. They adapt Kara to PagedAttention and introduce KvLLM, an inference framework built on vLLM, which reduces KV cache memory usage and improves output throughput.

Extensive experiments demonstrate consistent performance improvements for both Kara and KvLLM, indicating the method’s effectiveness in addressing the trade-offs between memory efficiency and model performance in reasoning LLM serving.