AdamX: Cosine similarity meets gradient descent
Researchers propose AdamX, a cosine-similarity-based first-order optimizer with variance rectification that matches Adam-class convergence across benchmark datasets and architectures.
The paper introduces AdamX, a first-order optimizer that uses cosine similarity as an adaptive mechanism for controlling update magnitudes, plus a variance rectification scheme for smoother optimization early in training. The method is described as scalable, model-agnostic, and straightforward to integrate into existing pipelines. Empirically, AdamX shows competitive convergence rates measured by epochs to reach performance thresholds under a fixed hyperparameter budget, with code and experiments released on GitHub.
- AdamX uses cosine similarity to adapt update magnitudes
- Variance rectification smooths early-stage optimization
- Claims competitive convergence across datasets and architectures
- Code released at github.com/FranciscoCaldas/adamX
Full article95 words · extracted from arxiv.org · click to collapse
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.11867