E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models
E-MoE adds a discrete MoE latent to diffusion language models and improves few-step generation.
E-MoE builds the reverse process of masked diffusion language models as a mixture of factorized distributions over a discrete latent given by Mixture-of-Experts routing. It does not increase active parameters over a factorized baseline and is meant to avoid posterior collapse of continuous Gaussian latents. Few-step generation improves on synthetic multi-modal benchmarks, binarized MNIST, and LM1B.
- Discrete MoE routing latent captures cross-position correlations
- No increase in active parameters over the factorized baseline
- Avoids posterior collapse of continuous Gaussian VAE latents
- Improves few-step generation on synthetic data, binarized MNIST, and LM1B
Full article127 words · extracted from huggingface.co · click to collapse
Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose Enhanced Mixture-of-Experts (E-MoE), which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline. Across synthetic multi-modal benchmarks, binarized MNIST, and LM1B, E-MoE improves few-step generation over factorized baselines.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.37533