QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
QuantMLA jointly quantizes MLA content and RoPE KV caches to INT4 with little accuracy loss and large throughput gains.
Multi-Head Latent Attention still grows KV-cache memory linearly with context and batch size, and RoPE-path quantization errors amplify strongly. QuantMLA learns offline, parameter-fusible transformations that reconstruct attention output on the content path and positional QK logits on the RoPE path. Across four MLA families it reports the first joint INT4 caching of both caches with minimal degradation; INT2 content cache plus INT4 RoPE keys remains competitive on reasoning and code. A native low-bit kernel delivers 3.59x compression at 128K context and 5.168x higher output throughput than BF16 under cache pressure.
- First reported joint INT4 caching of MLA content and RoPE paths.
- INT2 content cache with INT4 RoPE keys stays competitive.
- Path transforms fuse offline into weights with no online overhead.
- 128K context sees 3.59x compression and 5.168x throughput versus BF16.
From the original article
1 min read at huggingface.co
Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memory still scales linearly with context length and batch size.
By Zunhai Su, Yuxuan Sun, Jianchao Tan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.