Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
SplitMoE divides video-diffusion experts into semantic and generic roles, avoiding uniform routing and improving generation quality.
SplitMoE is a sparse architecture for scaling video diffusion models that rejects uniform expert balancing. It splits the expert pool into semantic experts, which capture high-level attributes, and generic experts, which preserve residual visual detail, using prototype-guided routing and pull-push regularization. Under an equivalent activated-parameter budget it outperformed traditional load-balanced mixture-of-experts models in convergence speed, routing coherence, and video generation quality, and the authors report an emergent coarse-to-fine denoising pattern.
- Uniform expert-usage regularization scatters coherent video patches across unrelated experts.
- SplitMoE splits the pool into semantic experts and generic residual experts.
- Prototype-guided routing and pull-push regularization cluster tokens by semantics.
- At equal activated parameters it converges faster and improves video quality.
From the original article
1 min read at huggingface.co
Mixture-of-Experts (MoE), popularized by large language models, is a promising paradigm for scaling visual generative models. However, conventional token-wise MoE routes tokens independently within a homogeneous expert pool and regularizes expert usage toward uniformity, making it poorly matched to video data that is spatiotemporally redundant and semantically long-tailed.
By Yu Xu, Yuxin Zhang, Xiao Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.