Pretraining data scale reverses Muon-AdamW fine-tuning gap
Original title: Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap
Beyond large pretraining budgets AdamW outperforms Muon after fine-tuning with the gap widening as model size grows.
Muon offers speedup over AdamW in minimizing pretraining loss but its advantage disappears after fine-tuning when data budgets are large. As model size increases AdamW overtakes Muon at progressively smaller token-parameter ratios. The reversal is explained via the margin-to-spread ratio measuring logit separation relative to spread in language models.
- AdamW outperforms Muon post fine-tuning
- Gap appears at large pretraining budgets
- Scale-dependent crossover point
- Explained by margin-to-spread ratio analysis
From the original article
1 min read at arxiv.orgMuon has recently emerged as a promising optimizer for pretraining large language models, offering a speedup over AdamW in minimizing pretraining loss. However, it remains unclear whether Muon's advantage persists after fine-tuning.
By Catherine Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.