Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Researchers trace unphysical video-diffusion motion to RoPE attention decay and propose a frequency-scaling fix.
The paper studies why video diffusion models produce physically implausible motion by inspecting attention during early denoising. It links motion planning to a subset of attention heads and finds that Rotary Position Embedding causes excessive spatial attention decay, locking early candidate regions into bad positions. The authors scale RoPE frequency across denoising steps so the model can explore more coherent trajectories. Both training-free and training-based tests are reported to improve physical commonsense without external physics priors.
- The study traces motion planning in text-to-video diffusion to early denoising attention heads.
- RoPE is found to cause excessive spatial attention decay and lock implausible trajectories.
- A lightweight change scales RoPE frequency across denoising steps.
- Training-free and trained experiments improve physical commonsense in generated video.
Full article188 words · extracted from huggingface.co · click to collapse
Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models. Specifically, we present the first interpretability study on the ''motion planning'' process of text-to-video diffusion models, revealing how motion trajectories form during early denoising stages. Building upon the ''first shape, then details'' finding, we combine cross-attention trajectory patterns with causal head contributions to identify a specific subset of attention heads driving motion planning. Further, our self-attention analysis shows that Rotary Position Embedding (RoPE) induces excessive spatial attention decay. This causes early candidate regions to prematurely lock into physically implausible positions, suppressing reasonable trajectories in adjacent frames and triggering generation failure modes. To address this fundamental flaw, we propose a lightweight architectural modification that scales the frequency of RoPE across different denoising steps. This strategy reduces excessive attention decay, helping the model explore better candidate regions to establish coherent physical motion. Finally, training-free and training-based experiments confirm the effectiveness of our approach in enhancing the physical commonsense of generated videos.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.23658