LoopVQ sub-1-bit KV cache quantization for looped transformers
Original title: LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers
LoopVQ jointly quantizes channels across multiple loop steps in looped transformers to reduce memory while preserving accuracy.
Looped transformers reuse layers for parameter efficiency but KV cache memory grows linearly with loop steps. LoopVQ codes channels jointly and fine-tunes codebooks to match full-precision outputs. At 0.5 bits per element it outperforms existing methods on general long chain-of-thought and retrieval tasks. Combined with TaSQ it retains BF16 accuracy at 0.25 bits and nearly halves decode latency in SGLang.
- Highly redundant KV cache along loop axis
- Joint channel coding for multiple loop steps
- 0.5 bits per element outperforms existing methods
- Retains accuracy at 0.25 bits with TaSQ
- Halves decode latency on 32K context
- Products
- SGLang
From the original article
1 min read at huggingface.co
Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. By reapplying the same layers across multiple loop steps, recent looped language models match the performance of non-looped models several times their size.
By Minsoo Cheong, Sungjoo Yoo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.