SoftServe: A Scalable Quasi-Newton Method for Deep Learning
SoftServe scales positive-definite quasi-Newton updates to large networks and beats Adam, Muon, and SOAP on ill-conditioned tasks.
SoftServe is a family of quasi-Newton optimizers for deep learning that estimates positive-definite curvature from a variational objective even when negative curvature is present, without line searches or ad hoc corrections. Diagonal and Kronecker-factored variants preserve positive definiteness by construction and use stable coupled Newton-Schulz iterations so matrix work stays as GPU-friendly multiplications. On severely ill-conditioned problems, including recurrent networks, deep autoencoders, physics-informed networks, and a 136-million-parameter physics-informed diffusion model, SoftServe often reaches lower loss than Adam, Muon, and SOAP.
- SoftServe builds positive-definite quasi-Newton curvature without line searches.
- Diagonal and Kronecker-factored variants scale to very large networks.
- Coupled Newton-Schulz iterations replace matrix decompositions with GPU multiplications.
- It beats Adam, Muon, and SOAP on ill-conditioned tasks, including a 136M diffusion model.
From the original article
1 min read at arxiv.orgQuasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections.
By Joohwan Ko · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.