Skip to content
NewA calmer, faster ZeroHour

CertiSum keeps MXFP4 training bitwise identical across GPUs and layouts

Original title: CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts

arXiv cs.AI / cs.LG / cs.CLby Steve Rhyner ()1 min read
AI summary · x-ai/grok-4.7Summaries by Vani AI

Two-hundred-step Qwen3-8B-Base states matched on AMD MI350X and NVIDIA GB200, with up to 60.5% added latency.

MXFP4 training can produce different results across parallelism layouts and GPU vendors. CertiSum certifies when native FP32 accumulation is already exact, recomputes uncertified outputs, and merges partial sums without rounding. In 200-step Qwen3-8B-Base experiments, training states stayed bitwise identical across AMD MI350X and NVIDIA GB200 and across data, tensor, and context parallelism on MI350X. Across three 8B–14B models it added 11.8–60.5% step latency versus non-exact MXFP4 while remaining faster than BF16.

Key facts
  • Speedups reached 1.24 times over BF16 and 1.70 times over deterministic BF16.
  • The broader runs used FineWeb-Edu on MI350X.
  • Quantization blocks remain unchanged across layouts.
  • Exact kernels are used where certification rarely succeeds.
  • Attention and weight-gradient partial sums are merged without rounding.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at arxiv.org

Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors.

arXiv cs.AI / cs.LG / cs.CL

By Steve Rhyner · arxiv.org

Read the full article at arXiv cs.AI / cs.LG / cs.CL

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.