Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent
Quantile-k-Loss SGD filters corrupted component losses by quantile sampling, proving linear convergence while outperforming standard and min-k-loss SGD.
The paper proposes Quantile-k-Loss SGD (Q(k)L-SGD), a loss-filtering framework for finite-sum optimization with corrupted components that samples k losses per iteration and updates using an index from the lower empirical q-quantile. The authors prove linear convergence under standard convexity, requiring sample size to scale with the number of corruptions, plus a complementary small-sample probabilistic analysis. Experiments on polynomial regression, regularized logistic regression, and hinge loss show intermediate quantiles often outperform both standard SGD and min-k-loss SGD.
- Quantile-k-Loss SGD samples k losses and updates from lower q-quantile
- Linear convergence proven under convexity with corruption-dependent sample size
- Small-sample analysis covers any sample size k probabilistically
- Intermediate quantiles outperform standard SGD and min-k-loss SGD on regression tasks
From the original article
1 min read at arxiv.orgWe study loss-based filtering for finite-sum optimization with a subset of corrupted component functions whose gradients may be highly unreliable.
By Jamie Haddock · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.