How to Loop MoE: Flatten the Experts, Untie the Attention
Foil flattens experts and unties attention in looped MoE models, lowering pretraining loss at equal parameters and compute.
Stories per day · 30 days
2 stories, newest first · one card per story
Stories tagged #foil
How to Loop MoE: Flatten the Experts, Untie the Attention
Foil flattens experts and unties attention so looped mixture-of-experts models train with lower loss.