CertiSum keeps MXFP4 training bitwise identical across GPUs and layouts
Original title: CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts
Two-hundred-step Qwen3-8B-Base states matched on AMD MI350X and NVIDIA GB200, with up to 60.5% added latency.
MXFP4 training can produce different results across parallelism layouts and GPU vendors. CertiSum certifies when native FP32 accumulation is already exact, recomputes uncertified outputs, and merges partial sums without rounding. In 200-step Qwen3-8B-Base experiments, training states stayed bitwise identical across AMD MI350X and NVIDIA GB200 and across data, tensor, and context parallelism on MI350X. Across three 8B–14B models it added 11.8–60.5% step latency versus non-exact MXFP4 while remaining faster than BF16.
- Speedups reached 1.24 times over BF16 and 1.70 times over deterministic BF16.
- The broader runs used FineWeb-Edu on MI350X.
- Quantization blocks remain unchanged across layouts.
- Exact kernels are used where certification rarely succeeds.
- Attention and weight-gradient partial sums are merged without rounding.
- AI models
- Qwen3-8B-Base
From the original article
1 min read at arxiv.orgTraining with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors.
By Steve Rhyner · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.