BitDecoding-FP4 keeps native FP4 KV caches near BF16 accuracy
Original title: BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
On NVIDIA B300, Qwen3-4B and Qwen3-30B-A3B peak throughput is 2.4–3.0× BF16 KV, within 0.5 points using NVFP4.
BitDecoding-FP4 is an algorithm–system design for native FP4 key-value cache decoding in long-context LLM serving. It targets query logit error, E4M3 scale underflow, and key outliers, and fuses probability quantization with stacked query MMA on paged KV caches. The authors report staying within 0.5 points of BF16 KV accuracy with NVFP4 and 1 point with MXFP4 across dense and MoE models using GQA and MLA, including RL rollouts. Code is at github.com/OpenBitSys/BitDecoding-FP4.
Unconfirmed: Accuracy and throughput numbers are the authors' reported results, not independently verified.
- Dual-Residual Query Quantization cuts reported query logit error from 60% to 1%.
- NVFP4 stays within 0.5 points of BF16 KV; MXFP4 within 1 point.
- End-to-end tests used NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B.
- Peak serving throughput is 2.4–3.0 times BF16 KV in those tests.
- Vendors
- NVIDIA
- Products
- B300
- Organizations
- OpenBitSys
- AI models
- Qwen3-4BQwen3-30B-A3B
From the original article
1 min read at huggingface.coThe key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving.
By Dayou Du, Woon Yee Ng, Guangming Sheng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.