Data-Efficient Classifier Pretraining Through Diffusion Models
DiffPT uses diffusion pretraining to improve low-data classifiers, especially on medical images and time series.
DiffPT is a diffusion-model pretraining method for classifier backbones when domain-specific labeled data is scarce. It revives reconstruction-style pretraining and is compared with modern and traditional unsupervised and self-supervised methods on time series, natural images, and medical images. Contrastive pretraining is stronger on CIFAR-10 and Tiny ImageNet in low-data regimes, while DiffPT outperforms self-supervised baselines across low-data medical imaging and time-series tasks. The authors present it as a simple cross-modality option for data-efficient classifier pretraining.
- DiffPT pretrains a classifier backbone with a simple diffusion objective.
- Contrastive pretraining wins on CIFAR-10 and Tiny ImageNet in low-data settings.
- DiffPT beats self-supervised baselines on medical imaging and time series.
- Products
- CIFAR-10Tiny ImageNet
From the original article
1 min read at arxiv.orgMachine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time.
By Qinyu Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.