DP-μP transfers learning rates under differential privacy
Original title: Towards Hyperparameter Transfer for Differentially Private Optimization
Scaling rules cover DP-SGD, DP-Adam, and DP-Muon on MLP, ViT, and GPT-2.
The paper says standard Maximal Update Parametrization fails under differential privacy because high-dimensional noise has different spectral properties from low-rank gradients. The authors introduce DP-μP and an SNR condition for layer-wise learning-rate scaling, with rules for DP-SGD, DP-Adam, and DP-Muon. They report zero-shot learning-rate transfer across model widths on MLP, ViT, and GPT-2 that matched or exceeded individually tuned baselines while cutting tuning cost.
Unconfirmed: Transfer gains are the authors’ measurements; no independent replication is stated.
- Standard μP failed under DP because noise and gradients differ spectrally.
- An SNR condition governs layer-wise learning-rate scaling.
- Evaluated architectures were MLP, ViT, and GPT-2.
- Authors say transfer matched or beat tuned baselines at lower compute cost.
From the original article
1 min read at arxiv.orgScaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping.
By Tianze Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.