LoopVQ enables sub 1-bit KV cache quantization for looped transformers
LoopVQ quantizes KV cache at 0.5 bits per element for looped transformers, nearly halving decode latency at 32K context.
Stories per day · 30 days
1 RSS
2 stories, newest first · one card per story
Stories tagged #memory-optimization
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Four scheduling techniques bound all memory peaks in long-context MoE training, enabling 120B-667B models at 1M-token context with up to 10.4x throughput.