GraphMuon optimizes graph neural networks with matrix-aware updates
Original title: Module-Aware Optimization for Graph Neural Networks
GraphMuon achieves 1.32x-2.42x speedups over AdamW on large OGB benchmarks.
Graph neural networks suffer from update amplification when applying matrix-aware optimizers to graph-propagated features. GraphMuon adapts Muon to the covariance structure of these representations with orthogonalization and normalization. It shows theoretical mitigation of amplification and empirical speedups across 15 dataset-architecture configurations.
- Lower time-to-target in 13 out of 15 configurations
- 1.32x-2.42x validation-loss speedups
- Competitive or superior predictive performance
- Code available on GitHub
From the original article
1 min read at arxiv.orgDifferent optimizers impose different geometries on parameter updates. Adaptive methods such as Adam and AdamW use coordinate-wise updates, while matrix-aware optimizers such as Muon utilize weight-matrix structure through orthogonalized updates.
By Guy Hadad · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.