On the Diffusibility of High-Dimensional Latents
Finetuning visual encoders for reconstruction shrinks latent dimensionality, and x0-prediction then improves text-to-image diffusion.
Representation autoencoders let diffusion models operate in pretrained visual-encoder feature spaces, but many off-the-shelf encoders discard fine detail. Finetuning those encoders for reconstruction recovers detail yet reduces effective dimensionality and changes geometry. Standard velocity prediction in flow matching then requires fitting orthogonal noise outside the low-dimensional signal manifold, which is inefficient. Across multiple strong-reconstruction encoders, clean-data x0-prediction consistently improves text-to-image generation by focusing learning on the signal manifold.
- Reconstruction finetuning lowers effective latent dimensionality
- Velocity prediction fits noise outside the signal manifold
- x0-prediction improves text-to-image generation across encoders
Full article127 words · extracted from huggingface.co · click to collapse
Representation Autoencoders (RAEs) enable diffusion models to operate in the feature spaces of pretrained visual encoders. However, many off-the-shelf encoders are not optimized for faithful reconstruction, discarding fine-grained visual details. As expected, finetuning these encoders for image reconstruction recovers such details. However, perhaps counterintuitively, this procedure reduces the effective dimensionality of the resulting representation, and the altered geometry has downstream effects on generation. Specifically, we show that using the standard velocity prediction in flow matching in this high-dimensional space requires the model to fit orthogonal noise directions outside the low-dimensional signal manifold, making optimization inefficient. This motivates using the clean data parameterization (x_{0}-prediction) instead, which focuses learning on the underlying signal manifold. Across experiments with multiple strong-reconstruction encoders, we show that x_{0}-prediction consistently improves text-to-image generation performance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.28473