Distributionally Robust Mixture-of-Experts Training
DRMoET improves imperfect-routing robustness, raising a 10.3B MoE model's seven-task average from 0.6625 to 0.6767.
1 stories, newest first · one card per story
DRMoET improves imperfect-routing robustness, raising a 10.3B MoE model's seven-task average from 0.6625 to 0.6767.
Today 1.6K readers · 2.5K views · 779 reads · all time 189K views