Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Study finds Mixture-of-Experts models overfit faster than dense Transformers under repeated training data, with degradation tied to total parameter sparsity.
Across models from 80M to 1B active parameters (8.5B total), MoE architectures degrade more rapidly than dense models when training data is repeated, with the effect increasing with sparsity as dictated by total parameters. Dense 80M models tolerate 8x repetition with minimal loss while MoEs suffer at 4x and underperform dense models beyond 32x. Masking-based regularization such as dropout mitigates overfitting, letting MoEs beat dense models even at over 64x repetition, though no method matches all-unique training data. Routing stabilizes early and expert specialization correlates with overfitting to repeated data.
- MoE degradation under repetition increases with sparsity, driven by total rather than active parameters.
- Dense 80M models repeat data 8x safely; MoEs begin degrading at 4x repetition.
- Strong masking regularization lets MoEs outperform dense models beyond 64x repetition.
- Expert specialization correlates with overfitting; routing stabilizes early in training.
From the original article
1 min read at arxiv.orgAs the supply of human-written text is exhausted, it has become standard practice to repeat language model training data. Prior work has studied data repetition for densely activated Transformers, but the effects of data repetition remains largely unexplored for recently dominant sparse architectures such as Mixture-of-Experts (MoE), despite their increased compute efficiency.
By Atindra Jha · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.