Multi-Epoch Pretraining Scaling Theory Shows Ensembling Benefits
Original title: A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
Logit averaging in ensembles reduces late-training overfitting and delays optimal stopping in fixed-data pretraining.
Pretraining is shifting to data-bound regimes. The authors examine multi-epoch training and ensembling on FineWeb. Ensembling lowers prediction variance, alters stopping times, and supports compute-allocation laws where ensemble size scales with compute. Excess test loss declines as a power of total compute.
- Ensembling delays optimal stopping
- Logit averaging outperforms wider single models
- Ensemble size grows with compute
- Excess test loss decreases as a power of compute
From the original article
1 min read at arxiv.orgIn the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed.
By Zhen Yang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.