LoopVQ enables sub 1-bit KV cache quantization for looped transformers
LoopVQ quantizes KV cache at 0.5 bits per element for looped transformers, nearly halving decode latency at 32K context.
Stories per day · 30 days
1 stories, newest first · one card per story