Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
Fixed-budget diffusion retraining loses only fragile features, unlike full replacement with synthetic data.
The paper studies model collapse in diffusion models when each generation is trained on a fixed-size sample drawn from a pool that keeps all past data and the original real data, so the real fraction shrinks without deletion. Full replacement of real data with synthetic data degrades a 2D spiral and MNIST, Fashion-MNIST, and CIFAR-10 rapidly, while the fixed-budget protocol degrades only partially. A linear-response model of multi-generation parameter dynamics shows fragile features are lost under both protocols, whereas robust features can persist for practically unbounded generations under the fixed budget.
- Replacement of real data with synthetic data collapses diffusion datasets quickly.
- Fixed-budget sampling from a growing mixed pool degrades only some features.
- Tested on a 2D spiral, MNIST, Fashion-MNIST, and CIFAR-10.
- A linear-response recursion separates fragile features from ones preserved under fixed budgets.
From the original article
1 min read at arxiv.orgModel collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.)
By Hanna Malet · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.