Looping cuts training FLOPs for uniform diffusion language models
Original title: Looped Uniform Diffusion Language Models
Looped UDLMs match a roughly 0.5-trillion-token baseline with up to 2.8x fewer FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu.
The paper studies selective looping inside uniform diffusion language models while leaving the training objective and sampler unchanged. Looping recursively applies a few early Transformer layers within each denoising step. Looped models reach the validation negative log-likelihood of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8 times fewer training FLOPs on OpenWebText and 3.3 times fewer on FineWeb-Edu. At a matched NFE budget they reach up to 9.8% lower generation perplexity, and sampling smaller loop counts more often uses 12% fewer loops downstream without hurting accuracy.
- Most generation-perplexity gains occur at moderate loop counts.
- Preferring smaller loop counts cuts downstream loops by 12% without accuracy loss.
- The training objective and sampler are left unchanged.
From the original article
1 min read at arxiv.orgUniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context.
By Saiteja Utpala · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.