AdamX: Cosine similarity meets gradient descent
Researchers propose AdamX, a cosine-similarity-based first-order optimizer with variance rectification that matches Adam-class convergence across benchmark datasets and architectures.
The paper introduces AdamX, a first-order optimizer that uses cosine similarity as an adaptive mechanism for controlling update magnitudes, plus a variance rectification scheme for smoother optimization early in training. The method is described as scalable, model-agnostic, and straightforward to integrate into existing pipelines. Empirically, AdamX shows competitive convergence rates measured by epochs to reach performance thresholds under a fixed hyperparameter budget, with code and experiments released on GitHub.