Prime Intellect ships trainable fused Mega MoE kernel for faster training
Original title: Prime Intellect Releases Prime Mega MoE: Trainable Fused MoE Kernel, 3.11x Faster Forward Pass
Self-reported full steps on Qwen3 MoE models were 32% to 34% faster than prime-rl and up to 55% faster than TorchTitan.
Prime Intellect released Prime Mega MoE, adding an overlapping backward pass to DeepSeek's fused Mega MoE kernel in DeepGEMM so it can train, not only infer. It runs on NVIDIA Hopper and Blackwell in BF16, with blockwise FP8 on Hopper and MXFP8 on Blackwell. Published timings used 8×H200 nodes. The largest single-layer gain was a 3.11x forward pass on Qwen3-30B-A3B; backward gains were smaller.
Unconfirmed: All speed figures are self-reported by Prime Intellect, and the TorchTitan comparison uses a different setup.
- Qwen3-30B-A3B steps fell from 2.78s to 2.08s versus the prime-rl baseline.
- Qwen3-235B-A22B steps fell from 12.72s to 9.65s on 128 H200s.
- Throughput versus TorchTitan rose from 5,337 to 7,886 tokens/s/GPU on the 30B model.
- Kernels ship in a prebuilt DeepGEMM wheel; stock DeepGEMM lacks the backward kernel.
- License for prime-rl is Apache 2.0.
- Vendors
- Prime IntellectDeepSeek
- AI models
- Qwen3-30B-A3BQwen3-235B-A22B
From the original article
7 min read at marktechpost.comPrime Intellect released Prime Mega MoE, a backward pass for DeepSeek’s fused Mega MoE kernel. It makes the kernel trainable and speeds Qwen3 MoE training steps by up to 34% in prime-rl. The post Prime Intellect Releases Prime Mega MoE: Trainable Fused MoE Kernel, 3.
By Asif Razzaq · marktechpost.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.