Structured Residual Connectivity Matters for Diffusion Transformers
Structured residual paths in diffusion transformers cut FID and training iterations with almost no extra parameters.
The paper redesigns residual connections in Diffusion Transformers so each block can selectively retrieve earlier spatial and semantic features instead of summing a single residual stream. The structured local and long-range paths add under 0.1% parameters and cut training iterations by up to 1.73×. Applied to REPA-XL/2, FID falls from 5.9 to 4.34 without guidance and reaches 1.39 with classifier-free guidance.
- DiTs use a uniform residual stream unlike U-Net skip connections.
- Structured local and long-range paths let blocks retrieve earlier features.
- Training needs up to 1.73× fewer iterations with under 0.1% extra parameters.
- REPA-XL/2 improves from 5.9 to 4.34 FID, and 1.39 FID with guidance.
Full article227 words · extracted from huggingface.co · click to collapse
Diffusion Transformers (DiTs) have established themselves as a scalable backbone for high-fidelity image synthesis. However, unlike U-Net based diffusion models that rely on rigid, hand-crafted skip connections, DiTs predominantly use a uniform residual stream that integrates all preceding layers as a monolithic state. In this work, we rethink residual connections in diffusion transformers and propose to transform them from passive summation into an active retrieval mechanism optimized for image denoising. First, we conduct a systematic analysis of DiT's internal representation, revealing a latent preference for early-layer feature reuse and symmetric layer guidance. Motivated by this, we introduce a structured connectivity design that explicitly integrates local residual connections with long-range pathways. Instead of static skip connections or dense all-layer routing, our method enables each transformer block to selectively ``attend'' to critical earlier representations, dynamically retrieving spatial and semantic cues through direct, differentiable cross-depth paths. Experiments show that our adaptive connectivity leads to faster convergence, with up to 1.73times fewer training iterations, and significant gains in FID and visual quality with less than 0.1% additional parameters, further improving a strong REPA-XL/2 model from 5.9 to 4.34 FID without guidance and reaching 1.39 FID with classifier-free guidance. Our findings suggest that adaptive cross-layer connectivity is a critical yet underexplored factor in diffusion transformers, and that incorporating structured information pathways provides a simple and effective direction for improving scalable generative models.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.33203