MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
MeqMuon balances row and column updates to improve LLM pretraining over AdamW and Muon.
MeqMuon is a matrix-equilibrating variant of the Muon optimizer for large language model pretraining. It balances row and column update magnitudes with normalization that adapts to different imbalance patterns without manual tuning. It also removes the need to store AdamW second-moment estimates, lowering optimizer-state memory. Experiments report better convergence than AdamW, Muon, and other baselines.
- Normalizes both row and column magnitudes without manual pattern selection.
- Drops AdamW second-moment storage, reducing optimizer-state memory.
- Reports better LLM pretraining convergence than AdamW and Muon.
Full article124 words · extracted from arxiv.org · click to collapse
The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and training efficiency in LLM pretraining. Recent work introduces row-wise normalization into Muon to balance update magnitudes and improve pretraining performance. However, row-wise normalization alone cannot accommodate different imbalance patterns in update matrices. In this paper, we propose an improved Muon optimizer, called \underline{m}atrix-\underline{eq}uilibrating Muon~(MeqMuon), for LLM pretraining. MeqMuon balances both row and column magnitudes through normalization that can be automatically tailored to different imbalance patterns without manual intervention. Moreover, MeqMuon eliminates the need to store AdamW's second-moment estimates, reducing optimizer-state memory usage. Empirical results demonstrate that MeqMuon achieves better convergence performance than AdamW, Muon, and other baselines in LLM pretraining.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.35701