DCKV learns end-to-end compression of hybrid-model prompt KV caches
Original title: DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models
On Qwen3.5-2B, tokens use over 55 times fewer bytes than bf16 with at most 1.8 points lost on three long-context benchmarks.
DCKV is a differentiable codec for the key-value cache kept by the few full-attention layers of hybrid language models. Its transform, quantizer, and entropy models, conditioned on prompt tokens and already decoded neighboring positions, are trained together against a frozen language model for both code length and output change. On Qwen3.5-2B, the paper reports more than 55 times fewer bytes than bf16 at a loss of at most 1.8 points on three long-context benchmarks, and more than 100 times fewer bytes at a loss of at most 3.6 points.
- The codec is trained end to end against a frozen language model.
- Entropy models are conditioned on prompt tokens and decoded neighbors.
- A tighter setting exceeds 100 times fewer bytes at up to 3.6 points lost.
From the original article
1 min read at arxiv.orgHybrid language models keep a key-value (KV) cache only in their few full-attention layers, but this cache is still the part of a prompt's state that grows with the prompt and that must be stored or moved to reuse it. We present DCKV, a differentiable codec for the KV cache and the first KV-cache codec that is trained end to end.
By Egor Maximov · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.