Paper unifies pretraining, SFT, and RL under one likelihood algorithm
Original title: Training Foundation Models with a Single Algorithm
The authors recast RL stages as maximum-likelihood fits to model-generated targets and mix them in one trainer.
The paper argues staged foundation-model training is not a fundamental algorithmic split. It derives RL objectives as maximum-likelihood fitting to model-generated target distributions, so pretraining, midtraining, SFT, and RL sit under one likelihood-based algorithm. A unified trainer mixes static data and RL generations. Across controlled and real-world experiments, the authors report a better accuracy-compute tradeoff, broader task coverage, preserved capabilities, and a stronger start for continual learning.
Unconfirmed: The paper does not name the models, datasets, or quantitative gains behind the reported improvements.
- Conventional staged training is treated as a special case of data choice and timing.
- The system mixes static data and RL generations in a single trainer.
- Authors say unified training improves the accuracy-compute tradeoff and continual-learning start.
From the original article
1 min read at huggingface.co
Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental.
By Yuda Song, Alvin Zhang, Fahim Tajwar · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.