Pretraining data scale reverses Muon-AdamW fine-tuning gap
Beyond large pretraining budgets AdamW outperforms Muon after fine-tuning with the gap widening as model size grows.
Stories per day · 30 days
3 RSS
3 stories, newest first · one card per story
Stories tagged #optimizers
MuonGS relaxes weight-norm control for LLM pretraining
On dense models it reports about 15% token-equivalent speedup versus MuonH at matched validation loss.
AdamX: Cosine similarity meets gradient descent
Researchers propose AdamX, a cosine-similarity-based first-order optimizer with variance rectification that matches Adam-class convergence across benchmark datasets and architectures.