AdamX: Cosine similarity meets gradient descent
Researchers propose AdamX, a cosine-similarity-based first-order optimizer with variance rectification that matches Adam-class convergence across benchmark datasets and architectures.
The paper introduces AdamX, a first-order optimizer that uses cosine similarity as an adaptive mechanism for controlling update magnitudes, plus a variance rectification scheme for smoother optimization early in training. The method is described as scalable, model-agnostic, and straightforward to integrate into existing pipelines. Empirically, AdamX shows competitive convergence rates measured by epochs to reach performance thresholds under a fixed hyperparameter budget, with code and experiments released on GitHub.
- AdamX uses cosine similarity to adapt update magnitudes
- Variance rectification smooths early-stage optimization
- Claims competitive convergence across datasets and architectures
- Code released at github.com/FranciscoCaldas/adamX
From the original article
1 min read at arxiv.orgWe introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training.
By Francisco Caldas · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.