LowRAM compresses KV cache with low-rank keys and amortised values
Original title: LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
Test-time value predictors plus cross-layer key compression give about 4× compression and up to 5× decode throughput.
The paper finds that values in each attention head are largely predictable from their keys and fits a lightweight per-head predictor at test time, storing residuals only for the worst reconstructions. Paired with cross-layer low-rank key compression, LowRAM stays near lossless across three model families and can raise peak decode throughput by up to 5× at 4× compression. It also composes with quantisation and eviction, reaching 26.9× compression at 94.5% of baseline performance on long contexts.
- Value amortisation alone gives 4× near-lossless value compression.
- Predictors can be meta-learned or fitted from scratch.
- Decoding reconstructs only a selected subset of KV pairs.
- With quantisation and eviction, compression reaches 26.9× at 94.5% of baseline.
From the original article
1 min read at huggingface.co
Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys.
By Teresa Delgado, Hugo Hazard, Martin Benfeghoul · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.