Skip to content
NewA calmer, faster ZeroHour

RoCoKV compresses reasoning-model KV cache inside vLLM

Original title: RoCoKV: Accelerating LLM Inference Serving through Uncertainty and Redundancy Aware KV Cache Compression

arXiv cs.AI / cs.LG / cs.CLby Payman Behnam ()1 min read1
Part of a story covered by 2 sourcesRoCoKV compresses reasoning-model KV cache inside vLLMMerged summary & timeline
AI summary · x-ai/grok-4.7Summaries by Vani AI

On DeepSeek-R1-Distill-Llama-8B, the vLLM plugin reports 1.27X speedup and 38% higher throughput on AIME25.

RoCoKV compresses the KV cache of large reasoning models, targeting long generated traces rather than prompts. Uncertainty-guided pruning changes compression with model confidence, while redundancy-aware selection down-weights tokens similar to recent reasoning. Fused CUDA decode kernels are integrated into vLLM as a plugin that keeps paged KV layout and CUDA graph capture. Authors report higher accuracy than prior methods under constrained budgets, plus a 1.43X speedup on a long synthetic benchmark.

Unconfirmed: Speedups and accuracy gains are author-reported, not independent measurements.

Key facts
  • Looping, self-verification, and corrections can leave redundant tokens with high attention.
  • More cache is kept during uncertain phases and pruning is stronger when confidence is high.
  • Versus Full-KV, AIME25 shows 1.27X speedup and 38% higher throughput.
  • The long synthetic benchmark shows 1.43X speedup and 45% higher throughput.
  • Experiments cover three reasoning benchmarks and two reasoning models.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

KV cache compression is critical for large reasoning models, where most of the cost comes from long generated reasoning traces rather than the input prompt. These traces often contain redundant and low-utility tokens due to looping, self-verification, and corrections. However, such tokens may still receive high attention scores, making it difficult to remove them without harming the reasoning process.

arXiv cs.AI / cs.LG / cs.CL

By Payman Behnam · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.