Calibrated distillation consolidates specialists in dynamic MoE models
Original title: Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution
On Turing-20B-A2B the method scores 0.9988 across four domains and beats Qwen3-8B on math and code.
The paper studies post-training for dynamically routed mixture-of-experts models, combining calibrated multi-teacher on-policy distillation with bounded-width dispatch, final-route replay, and overlapped rollout and training. On Turing-20B-A2B, the selected model reaches a four-domain normalized integration score of 0.9988 and keeps at least 90.16% of measured specialist gain in every domain. At a matched 8K response cap it exceeds shared SFT and Qwen3-8B on four reasoning-enabled mathematics and code benchmarks. A PPU profile cuts consolidation step time from 398.7 to 139.3 seconds.
Unconfirmed: Benchmark and speed figures are the authors' reported measurements only.
- Full calibration scores 0.9962 at K=32, versus 0.9724 without calibration.
- Unweighted math and code mean is 85.92, against 80.33 for Qwen3-8B and 74.21 for SFT.
- The model improves over SFT on ten of twelve broader metrics.
- Combined execution is reported 2.86 times faster than serial execution.
From the original article
1 min read at arxiv.orgWe study post-training for dynamically routed mixture-of-experts (MoE) models, focusing on integrating independently optimized specialists while keeping consolidation efficient.
By Yue He · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.