Designing Transformers for Continuous Diffusion Language Models
LangDiT redesigns Transformers for continuous diffusion LMs, cutting OpenWebText generation perplexity to 11.33–21.72.
The paper studies Transformer designs for continuous diffusion language models and introduces Language Diffusion Transformer (LangDiT). LangDiT combines feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. As a plug-in backbone it sets new generation-perplexity results: 21.72 GenPPL at 1024 NFEs on LangFlow and 11.33 GenPPL on Plaid-style formulations for 1024-token OpenWebText sequences.
- LangDiT adds feature and query-key normalization, wider noise conditioning, and confidence-gated self-conditioning.
- On OpenWebText, LangFlow with LangDiT reaches 21.72 GenPPL at 1024 NFEs and 44.72 at 128 NFEs.
- Plaid-style formulations reach 11.33 GenPPL at 1024 denoising NFEs and 27.51 at 128.
- Gains keep each model's representation, objective, and sampler unchanged.
Full article211 words · extracted from arxiv.org · click to collapse
Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.22330