BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
BeaconKV introduces training-free KV cache compression using beacon queries, cutting long-reasoning inference memory up to 5.8x while preserving accuracy.
The paper shows recency-based KV cache compression assumptions fail in long-horizon reasoning because Thought Revisiting Tokens (TRT) re-attend to distant context such as early task-solving plans. TRT queries cluster into a small number of similarity groups, which BeaconKV exploits by maintaining compact beacon query representatives to anticipate revisited KV pairs without storing full query history. The training-free method achieves up to 5.8x memory reduction and over 4.3x throughput improvement across four open-source large reasoning models while nearly preserving full cache accuracy.
- Identifies Thought Revisiting Tokens that re-attend to distant earlier context
- TRT queries cluster into few similarity groups in embedding space
- Training-free method needs no model retraining or fine-tuning
- Up to 5.8x memory reduction and 4.3x throughput gain on four open-source LRMs
- Vendors
- Hugging Face
- Products
- BeaconKV
From the original article
1 min read at huggingface.co
Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces.
By Janghyeon Kim, Minsoo Kim, Kyuhong Shim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.