Learned Unmasking Policies Improve Diffusion Language Model Efficiency
Original title: What Makes a Good Unmasking Policy? A Controlled Study of Learned Unmasking in Diffusion Language Models
A 856K-parameter policy trained on trajectory-contraction targets achieves best accuracy-NFE frontier on LLaDA-8B-Instruct.
Masked diffusion language models commit predictions for parallel generation, but efficiency depends on unmasking policies. Supervised policies distill teacher decoding; analysis shows labeling predictions matching final output is overly permissive at low NFE while hidden states complement confidence. About 10% NFE reduction offsets policy overhead. The resulting policy accepts only commitments staying on reference trajectory and achieves best accuracy-NFE frontier plus highest throughput on GSM8K, competitive on MATH-500 and HumanEval.
Unconfirmed: Results based on tested settings and models
- 10% NFE reduction offsets policy overhead
- Policy uses confidence and hidden states
- Trained on trajectory-contraction targets
- Best accuracy-NFE frontier on LLaDA-8B-Instruct
- Competitive on GSM8K MATH-500 HumanEval
From the original article
1 min read at arxiv.orgMasked diffusion language models generate tokens in parallel, but their efficiency depends on which predictions are committed. Learned unmasking policies improve the speed--accuracy trade-off, yet differ in supervision, inputs, and capacity, obscuring which choices matter.
By Jinya Sakurai · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.