Expert-Space Exploration in MoE Reinforcement Learning (ESRL)

ESRL explores the MoE expert-routing space during RL post-training, improving Qwen3-30B-A3B Pass@1 by 3.2 points and Pass@8 by 4.5 points over GRPO with no extra sampling or compute.

ESRL (Expert-Space Exploration Reinforcement Learning) is an architecture-aware framework that treats expert routing in Mixture-of-Experts models as an additional source of rollout diversity during RL post-training. Perturbing expert routing increases rollout diversity similarly to higher decoding temperature, but naive perturbation degrades quality; ESRL therefore anchors high-confidence experts, restricts stochastic routing to a plausible candidate pool, adapts perturbation strength via router entropy, and replays recorded expert paths during policy optimization to fix routing mismatch. It achieves the best performance across top-K, top-1, and shared-expert routing backbones on mathematics, science, and code tasks. On Qwen3-30B-A3B it improves average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points respectively, without additional sampling or compute. The two reports — the Hugging Face daily papers listing (2026-09-10) and the arXiv cs.AI/cs.LG/cs.CL listing (2026-09-11) — are consistent and report identical figures.

• Paper title: 'Expert-Space Exploration in MoE Reinforcement Learning'; ESRL = Expert-Space Exploration Reinforcement Learning.
• Method: treats MoE expert routing as an additional source of rollout diversity during RL post-training, with an effect similar to raising decoding temperature, but naive routing perturbation degrades quality.
• Mechanisms: anchors high-confidence experts, restricts stochastic routing to a plausible candidate pool, adapts perturbation strength via router entropy, and replays recorded expert paths during policy optimization.
• Result on Qwen3-30B-A3B: +3.2 percentage points average Pass@1 and +4.5 percentage points average Pass@8 over GRPO.
• Achieves best performance across top-K, top-1, and shared-expert MoE routing backbones on mathematics, science, and code tasks.
• Gains come at no additional sampling or compute cost.
• Sources: Hugging Face daily papers entry dated 2026-09-10 and arXiv (cs.AI/cs.LG/cs.CL) entry dated 2026-09-11; both report identical figures with no disagreements.