On the Diffusibility of High-Dimensional Latents
Paper finds x0-prediction outperforms velocity prediction for diffusion in high-dimensional visual latents.
The paper studies Representation Autoencoders that let diffusion models generate in the feature space of pretrained visual encoders. Finetuning those encoders for faithful reconstruction restores fine detail but reduces the representation's effective dimensionality. Standard flow-matching velocity prediction then spends capacity fitting noise directions orthogonal to the low-dimensional signal manifold. Predicting clean data, x0-prediction, instead focuses learning on that manifold and consistently improves text-to-image performance across several strong-reconstruction encoders.
- Reconstruction finetuning lowers effective latent dimensionality.
- Velocity prediction fits noise outside the signal manifold.
- x0-prediction improves text-to-image generation across encoders.
- Experiments cover multiple strong-reconstruction visual encoders.
Full article127 words · extracted from arxiv.org · click to collapse
Representation Autoencoders (RAEs) enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reconstruction recovers such details. However, perhaps counterintuitively, this procedure reduces the effective dimensionality of the resulting representation, and the altered geometry has downstream effects on generation. Specifically, we show that using the standard velocity prediction in flow matching in this high-dimensional space requires the model to fit orthogonal noise directions outside the low-dimensional signal manifold, making optimization inefficient. This motivates using the clean data parameterization ($\boldsymbol{x}_{0}$-prediction) instead, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, we show that $\boldsymbol{x}_{0}$-prediction consistently improves text-to-image generation performance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.28473