Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
SplitMoE splits video-diffusion experts into semantic and generic roles, improving quality under a fixed parameter budget.
The paper argues that conventional visual mixture-of-experts models fall into a uniformity trap: independent token routing plus load-balancing regularization fragments spatiotemporally coherent video patches. SplitMoE splits the expert pool into semantic experts for high-level abstraction and generic experts for residual visual information, using prototype-guided routing and pull-push regularization. Under an equivalent activated-parameter budget it outperforms load-balanced MoEs on convergence speed, routing coherence, and video generation quality. The authors also report an emergent coarse-to-fine denoising pattern.
- Uniform token-wise MoE scatters coherent video patches across experts and distorts structure.
- SplitMoE separates semantic experts from generic experts that preserve residual visual detail.
- Prototype-guided routing and pull-push regularization replace forced uniform expert usage.
- At equal activated parameters it converges faster and improves video quality on standard benchmarks.
From the original article
1 min read at arxiv.orgMixture-of-Experts (MoE), popularized by large language models, is a promising paradigm for scaling visual generative models. However, conventional token-wise MoE routes tokens independently within a homogeneous expert pool and regularizes expert usage toward uniformity, making it poorly matched to video data that is spatiotemporally redundant and semantically long-tailed.
By Yu Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.