PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion
PixelDense separates semantic and geometric alignment teachers, improving pixel diffusion quality and training speed.
PixelDense uses dense-prediction models as representation-alignment targets for pixel-space diffusion, splitting DINOv2 and SAM2 into a semantic stream and Depth Anything v2 and Metric3D v2 into a geometric stream, with a weight-space orthogonality penalty. A flat sum of all four teachers underperformed the best geometric teacher because semantic and geometric gradients competed. Teachers are frozen during training and dropped at inference. On PixelGen and DeCo, PixelDense improved GenEval, DPG-Bench, and HPS v2.1, lifting PixelGen-XXL GenEval Overall from 0.7927 to 0.8093, with up to 53.1% PQ gain and 36.0% depth AbsRel reduction in partial-noise probes.
- Semantic and geometric teachers use separate projection streams plus an orthogonality penalty.
- All four teachers stay frozen in training and are removed at inference.
- PixelGen-XXL GenEval Overall rises from 0.7927 to 0.8093.
- From scratch, PixelDense hits the baseline peak GenEval 1.23x faster.
- SDEdit background PSNR on PIE-Bench improves by up to 2.2 dB.
From the original article
1 min read at huggingface.coRepresentation alignment (REPA) accelerates diffusion transformer training, but its alignment targets are almost exclusively semantic encoders such as DINOv2 and CLIP. Recent analysis points to spatial structure, not global semantics, as the carrier of the alignment effect, yet dense-prediction foundation models trained to predict that structure remain overlooked as REPA targets.
By Lehan Yang, Daiqing Qi, Wenhao Zhang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.