ALoDLM: Adaptively Looped Diffusion Language Models
ALoDLM allocates diffusion-model computation by token difficulty and beats matching autoregressive baselines.
ALoDLM replaces the uniform computation depth of diffusion language models with token-adaptive latent recurrence, spending more passes on harder unknown tokens and committing easy tokens as discrete context. Token-wise computation schedules are treated as latent variables and trained with a conditional negative evidence lower bound. At 1.7B and 8B parameters, ALoDLM reports a higher average score than evaluated diffusion models and matching autoregressive baselines across eleven benchmarks, while retaining parallel decoding.
- Uniform diffusion depth mismatches easy and hard unknown tokens.
- Ready tokens are committed while unresolved tokens keep refining latents.
- Models are trained at 1.7B and 8B parameters.
- Average score beats evaluated DLM and autoregressive baselines on eleven benchmarks.
From the original article
1 min read at huggingface.co
Diffusion language models (DLMs) enable fast generation by predicting multiple tokens in parallel, but their practical adoption remains limited by a persistent quality gap relative to comparably sized autoregressive (AR) models. We attribute this gap to a computation-difficulty mismatch: within a partially observed sequence, some unknown tokens are easy to predict, while others require substantially more computation.
By Liancheng Fang, Zhuowei Li, Youngeun Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.