Looped uniform diffusion language models cut training FLOPs
Original title: Looped Uniform Diffusion Language Models
Selective looping matches a non-looped UDLM's validation NLL with up to 3.3x fewer training FLOPs.
The paper studies selective looping inside uniform diffusion language models while leaving the training objective and sampler unchanged. Looped UDLMs reach the validation negative log-likelihood of a non-looped model trained on about 0.5 trillion tokens with up to 2.8x fewer FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. At a matched NFE budget, they report up to 9.8% lower generation perplexity, and sampling smaller loop counts more often uses 12% fewer loops on downstream tasks without lower accuracy.
- Efficiency gains are up to 2.8x FLOPs on OpenWebText and 3.3x on FineWeb-Edu.
- The non-looped comparison uses about 0.5 trillion tokens.
- Generation perplexity is up to 9.8% lower at matched NFE.
- Preferring smaller loop counts uses 12% fewer inference loops.
- The training objective and sampler are kept unchanged.
From the original article
1 min read at huggingface.co
Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context.
By Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.