STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
STEPQuant quantizes delta-rule recurrent states by error lifetime and location, matching FP32 accuracy at 6 bits.
STEPQuant is a post-training quantization method for delta-rule recurrent states used by linear attention. It allocates bits using how long memory lives and how strongly each key row affects output error, then jointly fits key-row and value-column scales. On Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, a nominal 6-bit budget closely matches FP32-state accuracy on long and short generation, while 4-bit STEPQuant beats uniform INT8. Integrated into SGLang, it compresses recurrent state more than 5x and cuts total serving memory by up to 68.7%.
- Quantization error persists across decoding steps and varies by key row and state magnitude.
- STEPQuant assigns precision by error size and memory lifetime and fits row and column scales.
- On Qwen3.8-27B and Kimi-Linear-48B, 6-bit STEPQuant matches FP32-state accuracy.
- Its 4-bit setting beats uniform INT8; SGLang kernels cut serving memory by up to 68.7%.
- Nominal 6-bit compression exceeds 5x for recurrent states.
- Products
- SGLang
From the original article
1 min read at huggingface.co
Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates.
By Bingchen Yao, Haobo Xu, Haokun Lin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.