LoopVQ enables sub 1-bit KV cache quantization for looped transformers
Original title: LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers
LoopVQ quantizes KV cache at 0.5 bits per element for looped transformers, nearly halving decode latency at 32K context.
Looped transformers reduce parameters by reapplying layers across multiple steps but still store keys and values per loop step. LoopVQ codes channels of multiple loop steps jointly and fine-tunes codebooks. At 0.5 bits per element it outperforms existing KV cache quantization methods on general, long chain-of-thought and long-context tasks. Combined with TaSQ it retains much of the BF16 accuracy even at 0.25 bits per element.
- 0.5 bits per element outperforms existing methods
- Retains BF16 accuracy at 0.25 bits per element with TaSQ
- Nearly halves decode latency at 32K-token context
- Raises peak serving throughput by up to 4.3x and 6.1x on 1.4B and 2.6B Ouro models
From the original article
1 min read at arxiv.orgLooped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match the performance of non-looped models several times their size.
By Minsoo Cheong · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.