Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Transfer from ridge-regression pretraining can achieve minimax-optimal scaling under power-law covariance shift.
The paper analyzes transfer learning in high-dimensional ridge regression, where a source model is pretrained and a target model is regularized by its L2 distance from the source estimator. It derives a deterministic equivalent for risk under model and covariance shift. For a shared regression function and commuting power-law covariance spectra, the resulting scaling law reaches the minimax-optimal exponent, and the optimal source regularization depends only on source data. The same rate is achieved by one-pass stochastic gradient descent, with experiments on synthetic data and language models.
- A ridge-regression source model is fine-tuned with L2 distance to that estimator.
- A deterministic risk equivalent covers both model shift and covariance shift.
- Under shared regression and power-law spectra, the scaling exponent is minimax-optimal.
- Optimal source regularization depends only on source data and transfers across targets.
- One-pass SGD fine-tuning achieves the same optimal rate.
From the original article
1 min read at arxiv.orgPretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law.
By Yalda Shabanzadeh · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.