Distributionally Robust Mixture-of-Experts Training
DRMoET improves imperfect-routing robustness, raising a 10.3B MoE model's seven-task average from 0.6625 to 0.6767.
Stories per day · 30 days
3 stories, newest first · one card per story
Stories tagged #sparse-models
How to Loop MoE: Flatten the Experts, Untie the Attention
Foil flattens experts and unties attention so looped mixture-of-experts models train with lower loss.
Learning Sparse Decision Trees via Transformer Variational Auto-Encoders
TREVIS uses a Tree Transformer VAE latent space to learn decision trees matching near-optimal predictive performance while improving structural sparsity.