Adversarial training with large step sizes shows persistent implicit bias
Original title: Adversarial Training with Large Step Sizes: Implicit Bias, Optimization, and Sharpness
Large constant step sizes in adversarial training preserve implicit bias gap with standard training and exhibit two-stage sharpness dynamics.
The paper studies deterministic gradient descent in adversarial training with large constant step sizes in linear logistic regression and neural networks. Implicit bias gap between AT and ST persists for arbitrarily large step sizes. No uniform convergence rate exists for optimization error in AT unlike standard training and sharpness shows two-stage dynamics.
- Implicit bias gap persists for arbitrarily large constant step sizes
- No uniform rate of convergence for optimization error in AT
- Sharpness exhibits two-stage dynamics unlike Edge-of-Stability in ST
From the original article
1 min read at arxiv.orgAdversarial Training (AT) is an important method for improving model robustness. We study the dynamics of deterministic Gradient Descent (GD) in AT. Most existing works in this direction focus on the small or vanishing step sizes regime, where GD resembles a continuous-time flow.
By Yi Feng · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.