Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Grouped Value Attention stores grouped values and reconstructs content keys via a learned linear map, cutting KV-cache size about 45-47% versus GQA.
GVA stores only grouped values and reconstructs content keys with a learned linear map absorbed into the query at decode time, while a small shared decoupled RoPE channel preserves positional information via a separately cached positional key. At 350M parameters trained on 30B FineWeb-Edu tokens, the 16-dimensional positional variant scores 44.18 average accuracy across five tasks versus 44.36 for GQA and 43.88 for MLA. Custom decoding kernels are in development with an open-source release planned.
- Replaces per-step key+value storage with grouped values plus on-demand key reconstruction.
- Reduces persistent cache scalars by ~45-47% relative to matched GQA configurations.
- Near-GQA accuracy: 44.18 vs 44.36 (GQA) and 43.88 (MLA) across five tasks at 350M scale.
- Shared decoupled RoPE channel retains positional information; custom decode kernels and open release planned.
From the original article
1 min read at huggingface.co
The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query attention (GQA) reduces this cost by sharing key-value heads, but still stores both a key and a value at every step. We introduce Grouped Value Attention (GVA), which stores grouped values and reconstructs content keys with a learned linear map.
By Vishesh Tripathi, Abhay Kumar, Ramsha Khan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.