Adam's implicit bias depends on learning rate schedule
Original title: Watch Your Step: Adam's Margin Maximization Bias Depends on the Learning Rate Schedule
Adam maximizes weighted l2 margin for constant schedules on homogeneous models.
We analyze Adam's implicit bias on homogeneous models in binary classification. For constant and weakly decaying learning rate schedules, Adam has no non-trivial trajectory-independent implicit regularizer. It maximizes a trajectory-dependent weighted-l2 margin. In intermediate regimes an l-inf bias occurs under gradient convergence conditions.
- No non-trivial locally Lipschitz convex regularizer for alpha > 1/L - 1/2
- Maximizes trajectory-dependent weighted-l2 margin
- Limiting parameter direction depends on initialization
- Loss decay rate depends only on second-moment parameter
From the original article
1 min read at arxiv.orgWe study the implicit bias of Adam and its similarity to sign gradient descent on homogeneous models in binary classification tasks. Perhaps surprisingly, we uncover that these depend on the learning rate schedule.
By Eitan Gronich · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.