TDLM reuses past denoiser states to speed diffusion language models
Original title: TDLM: Adaptive Temporal Integration for Diffusion Language Models
On LLaDA the added modules are about 0.84% of parameters and generation is up to 7.15 times faster.
TDLM adds temporal self-conditioning so unresolved tokens in a frozen diffusion-language-model denoiser can attend to their own earlier hidden states. Training uses self-generated trajectories and cross-entropy on tokens revealed between consecutive decoding steps, with causal temporal attention for parallel training. Across LLaDA and Dream, the paper reports up to 7.15 times faster generation than the original models and competitive answer quality on the evaluated tasks.
- Temporal modules add about 0.84% trainable parameters on LLaDA.
- The main comparison reports up to 7.15 times faster generation.
- Training uses self-generated trajectories and a cross-entropy objective.
- Results cover both LLaDA and Dream diffusion language models.
From the original article
1 min read at arxiv.orgMasked diffusion language models repeatedly compute representations for unresolved tokens, but conventional denoisers do not explicitly retain these representations for reuse across steps. We introduce Temporal Self-Conditioning Diffusion Language Model (TDLM), which lets each unresolved position attend to its own history of hidden representations within a frozen denoiser.
By Bohang Sun · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.