Differentially Private Mixing of Public Datasets Improves Private Learning
Private mixing of public datasets improves DP pretraining utility on NIH X-rays and ENRON text.
The paper introduces a pipeline that privately chooses a mixture of public datasets for pretraining before differentially private finetuning, by learning a low-dimensional linear model. On NIH ChestX-ray14, tailored X-ray mixtures improved macro AUC by up to 0.037 versus baselines, including a 22.8% relative AUC gain on Cardiomegaly at ε=1. For language modeling on the ENRON email corpus, pretraining on a mixture drawn from The Common Pile reduced test perplexity by 16% relative to baseline mixtures.
- Privately learns a low-dimensional mixture of public pretraining datasets.
- Macro AUC rose by up to 0.037 on NIH ChestX-ray14.
- Cardiomegaly relative AUC improved 22.8% at epsilon 1.
- ENRON test perplexity fell 16% versus baseline mixtures.
From the original article
1 min read at arxiv.orgMany machine learning applications involve sensitive data and therefore require training under differential privacy (DP). However, DP training often degrades model utility. In some cases, first pre-training the model on "public" data before finetuning with DP on the sensitive data can reduce the drop in utility.
By Yufei Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.