RoCoKV compresses reasoning-model KV cache inside vLLM
First merged summary.
RoCoKV, a vLLM plugin, compresses long reasoning traces and reports faster decode and higher throughput under cache limits.
RoCoKV compresses the KV cache of large reasoning models, targeting long generated traces rather than prompts. Uncertainty-guided pruning varies compression with model confidence, and redundancy-aware selection down-weights tokens similar to recent reasoning. Fused CUDA decode kernels ship as a vLLM plugin that keeps paged KV layout and CUDA graph capture. On DeepSeek-R1-Distill-Llama-8B, the authors report a 1.27X speedup and 38% higher throughput versus Full-KV on AIME25, and a 1.43X speedup and 45% higher throughput on a long synthetic benchmark. They also report higher accuracy than prior methods under constrained budgets across three reasoning benchmarks and two models.
- Targets generated reasoning traces, not prompts.
- Keeps more cache when uncertain; prunes harder when confident.
- vLLM plugin preserves paged KV layout and CUDA graph capture.
- AIME25: 1.27X speedup and 38% higher throughput versus Full-KV.
- Long synthetic benchmark: 1.43X speedup and 45% higher throughput.
- Tested on three reasoning benchmarks and two reasoning models.
Timeline
dated events in the reporting · oldest first- Nov 19, 2026RoCoKV paper and vLLM plugin results posted · arXiv
What each outlet adds
beyond the shared facts- arXivUncertainty and redundancy KV compression with vLLM speed and throughput figures
- Products
- vLLM
- AI models
- EchoMoEDeepSeek-R1-Distill-Llama-8B
Coverage
oldest first · one row per article · UTCEchoMoE cuts MoE expert-parallel traffic by sending compressed differences
The retrofit needs about 8,500 times fewer tokens than full pretraining and ships with a vLLM inference integration.
RoCoKV compresses reasoning-model KV cache inside vLLM
On DeepSeek-R1-Distill-Llama-8B, the vLLM plugin reports 1.27X speedup and 38% higher throughput on AIME25.