Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Grouped Value Attention stores grouped values and reconstructs content keys via a learned linear map, cutting KV-cache size about 45-47% versus GQA.
GVA stores only grouped values and reconstructs content keys with a learned linear map absorbed into the query at decode time, while a small shared decoupled RoPE channel preserves positional information via a separately cached positional key. At 350M parameters trained on 30B FineWeb-Edu tokens, the 16-dimensional positional variant scores 44.18 average accuracy across five tasks versus 44.36 for GQA and 43.88 for MLA. Custom decoding kernels are in development with an open-source release planned.