BitDecoding-FP4 enables accurate FP4 KV cache decoding
Original title: BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
Algorithm-system co-design achieves near-BF16 accuracy with 2.4-3x higher throughput for LLM serving.
BitDecoding-FP4 co-designs quantization for native FP4 KV decoding. It mitigates query logit, underflow and key outlier errors via dual-residual, probability scaling and smoothing techniques. Transposed softmax removes the bottleneck at the system level. Results hold across dense and MoE models on NVIDIA B300.
- Stays within 0.5 points of BF16 KV accuracy
- 2.4-3.0x higher peak serving throughput
- Works with Qwen3-4B and Qwen3-30B-A3B
- Available on GitHub
- Tracks BF16 in RL training rollouts
From the original article
1 min read at arxiv.orgThe key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving.
By Dayou Du · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.