On the Design Space of Continuous Diffusion Language Models
A study of continuous diffusion language models introduces DoubleCat and SemiCat, matching discrete diffusion on TinyGSM.
The paper examines self-conditioning and embedding geometry in continuous diffusion language models. It argues that self-conditioning corrects a train-inference mismatch, and introduces DoubleCat, a two-level interpolant that recovers those gains without an extra forward pass and improves TinyGSM accuracy more than threefold. Adding discrete reveals produces SemiCat, which reaches 49.94% on TinyGSM. At 1.7 billion parameters, SemiCat outperforms discrete diffusion on four of six likelihood evaluations and surpasses one-hot flows on unconditional generation.
- Self-conditioning corrects a mismatch between training and inference states.
- DoubleCat raises TinyGSM accuracy more than threefold without an extra pass.
- SemiCat reaches 49.94 percent on TinyGSM under standard hyperparameters.
- At 1.7B, SemiCat beats discrete diffusion on four of six likelihood tests.
From the original article
1 min read at arxiv.orgContinuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion.
By Oscar Davis · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.