Skip to content
NewA calmer, faster ZeroHour
StoryAI researchinfo

RoCoKV compresses reasoning-model KV cache inside vLLM

1 source · 2 articlesFirst reported Updated ()1
What's new

First merged summary.

Merged summary · x-ai/grok-4.7 · rewritten as coverage arrivesSummaries by Vani AI

RoCoKV, a vLLM plugin, compresses long reasoning traces and reports faster decode and higher throughput under cache limits.

RoCoKV compresses the KV cache of large reasoning models, targeting long generated traces rather than prompts. Uncertainty-guided pruning varies compression with model confidence, and redundancy-aware selection down-weights tokens similar to recent reasoning. Fused CUDA decode kernels ship as a vLLM plugin that keeps paged KV layout and CUDA graph capture. On DeepSeek-R1-Distill-Llama-8B, the authors report a 1.27X speedup and 38% higher throughput versus Full-KV on AIME25, and a 1.43X speedup and 45% higher throughput on a long synthetic benchmark. They also report higher accuracy than prior methods under constrained budgets across three reasoning benchmarks and two models.

Key facts
  • Targets generated reasoning traces, not prompts.
  • Keeps more cache when uncertain; prunes harder when confident.
  • vLLM plugin preserves paged KV layout and CUDA graph capture.
  • AIME25: 1.27X speedup and 38% higher throughput versus Full-KV.
  • Long synthetic benchmark: 1.43X speedup and 45% higher throughput.
  • Tested on three reasoning benchmarks and two reasoning models.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

Timeline

dated events in the reporting · oldest first
  1. Nov 19, 2026RoCoKV paper and vLLM plugin results posted · arXiv

What each outlet adds

beyond the shared facts
  • arXivUncertainty and redundancy KV compression with vLLM speed and throughput figures

Coverage

oldest first · one row per article · UTC
  1. · 3d ago arXiv cs.AI / cs.LG / cs.CLFirst report

    EchoMoE cuts MoE expert-parallel traffic by sending compressed differences

    The retrofit needs about 8,500 times fewer tokens than full pretraining and ships with a vLLM inference integration.

  2. · 1d ago arXiv cs.AI / cs.LG / cs.CL

    RoCoKV compresses reasoning-model KV cache inside vLLM

    On DeepSeek-R1-Distill-Llama-8B, the vLLM plugin reports 1.27X speedup and 38% higher throughput on AIME25.