Virtual Head Attention
Virtual Head Attention cuts KV cache versus GQA while improving quality and folding into standard GQA inference.
Virtual Head Attention uses Q Premix and a low-rank Linear Postmix so models can keep only two KV heads while recovering virtual head diversity and still folding into a standard GQA-2 architecture at inference. Across four scales pretrained on 50B tokens at 4K context, it beats GQA by 0.05 to 0.81 percentage points, cuts KV cache 4x versus GQA-8 at dense scales and 2x versus GQA-4 at MoE scale, and lowers training and prefill FLOPs by 7.9% on a 1.7B model. On an industrial 30B-A3B MoE trained on 500B tokens at 8K, it outperforms MLA and GQA-4 by 0.70 and 2.60 points. A matched 10B-A1B comparison scores 84.4 average RULER from 8K-128K versus 73.0 for GQA.
- Q Premix and Linear Postmix fold into standard GQA projections at inference.
- With two KV heads, VHA cuts KV cache 4x versus GQA-8 at dense scales.
- Across four scales, VHA beats GQA by 0.05 to 0.81 percentage points after 50B tokens.
- A 30B-A3B MoE gains +0.70% versus MLA and +2.60% versus GQA-4 on 12 benchmarks.
- Products
- Virtual Head Attention
From the original article
1 min read at huggingface.co
The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA can compress the cache, but MLA suffers from quality loss while MFA substantially increases training and prefill compute.
By Xibo Ding, Guoxia Wang, Jinle Zeng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.