Skip to content
NewA calmer, faster ZeroHour

BitDecoding-FP4 keeps native FP4 KV caches near BF16 accuracy

Original title: BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design

Hugging Face daily papersby Dayou Du, Woon Yee Ng, Guangming Sheng ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

On NVIDIA B300, Qwen3-4B and Qwen3-30B-A3B peak throughput is 2.4–3.0× BF16 KV, within 0.5 points using NVFP4.

BitDecoding-FP4 is an algorithm–system design for native FP4 key-value cache decoding in long-context LLM serving. It targets query logit error, E4M3 scale underflow, and key outliers, and fuses probability quantization with stacked query MMA on paged KV caches. The authors report staying within 0.5 points of BF16 KV accuracy with NVFP4 and 1 point with MXFP4 across dense and MoE models using GQA and MLA, including RL rollouts. Code is at github.com/OpenBitSys/BitDecoding-FP4.

Unconfirmed: Accuracy and throughput numbers are the authors' reported results, not independently verified.

Key facts
  • Dual-Residual Query Quantization cuts reported query logit error from 60% to 1%.
  • NVFP4 stays within 0.5 points of BF16 KV; MXFP4 within 1 point.
  • End-to-end tests used NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B.
  • Peak serving throughput is 2.4–3.0 times BF16 KV in those tests.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving.

Hugging Face daily papers

By Dayou Du, Woon Yee Ng, Guangming Sheng · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.