RoCoKV compresses reasoning-model KV cache inside vLLM
Original title: RoCoKV: Accelerating LLM Inference Serving through Uncertainty and Redundancy Aware KV Cache Compression
On DeepSeek-R1-Distill-Llama-8B, the vLLM plugin reports 1.27X speedup and 38% higher throughput on AIME25.
RoCoKV compresses the KV cache of large reasoning models, targeting long generated traces rather than prompts. Uncertainty-guided pruning changes compression with model confidence, while redundancy-aware selection down-weights tokens similar to recent reasoning. Fused CUDA decode kernels are integrated into vLLM as a plugin that keeps paged KV layout and CUDA graph capture. Authors report higher accuracy than prior methods under constrained budgets, plus a 1.43X speedup on a long synthetic benchmark.
Unconfirmed: Speedups and accuracy gains are author-reported, not independent measurements.
- Looping, self-verification, and corrections can leave redundant tokens with high attention.
- More cache is kept during uncertain phases and pruning is stronger when confidence is high.
- Versus Full-KV, AIME25 shows 1.27X speedup and 38% higher throughput.
- The long synthetic benchmark shows 1.43X speedup and 45% higher throughput.
- Experiments cover three reasoning benchmarks and two reasoning models.
- Products
- vLLM
- AI models
- DeepSeek-R1-Distill-Llama-8B
From the original article
1 min read at arxiv.orgKV cache compression is critical for large reasoning models, where most of the cost comes from long generated reasoning traces rather than the input prompt. These traces often contain redundant and low-utility tokens due to looping, self-verification, and corrections. However, such tokens may still receive high attention scores, making it difficult to remove them without harming the reasoning process.
By Payman Behnam · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.