Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Researchers trace unphysical video-diffusion motion to RoPE attention decay and propose a frequency-scaling fix.
The paper studies why video diffusion models produce physically implausible motion by inspecting attention during early denoising. It links motion planning to a subset of attention heads and finds that Rotary Position Embedding causes excessive spatial attention decay, locking early candidate regions into bad positions. The authors scale RoPE frequency across denoising steps so the model can explore more coherent trajectories. Both training-free and training-based tests are reported to improve physical commonsense without external physics priors.
- The study traces motion planning in text-to-video diffusion to early denoising attention heads.
- RoPE is found to cause excessive spatial attention decay and lock implausible trajectories.
- A lightweight change scales RoPE frequency across denoising steps.
- Training-free and trained experiments improve physical commonsense in generated video.
From the original article
1 min read at huggingface.co
Despite impressive visual quality, state-of-the-art video diffusion models often generate content that violates real-world physical laws. While existing solutions rely on external priors or specialized data, we investigate the root cause by exploring the internal mechanisms of these models.
By Yueyan Li, Haibo Wang, Caixia Yuan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.