Paper unifies foundation-model pretraining, SFT and RL under one algorithm
Original title: Training Foundation Models with a Single Algorithm
A single likelihood trainer mixes static data and RL generations and is reported to improve the accuracy-compute tradeoff.
The paper derives reinforcement-learning objectives as maximum-likelihood fits to model-generated target distributions, so pretraining, midtraining, supervised fine-tuning and RL share one likelihood-based algorithm. Staged training becomes a special case of choosing data, proportions and timing. A unified trainer mixes static data and RL generations. Across controlled and real-world experiments, the authors report a better accuracy-compute tradeoff, broader task coverage and a stronger start for continual learning.
- Stages covered are pretraining, midtraining, SFT and RL.
- Conventional staged training is treated as one data-schedule case.
- The system mixes static data and RL generations in one trainer.
- Authors say unified training preserves broader capabilities.
- No named model, dataset size or compute figure is given.
From the original article
1 min read at arxiv.orgFoundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental.
By Yuda Song · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.