Unmask the State: When Does State Adaptation Matter for Masked Diffusion Language Models
Selective state adaptation captures most masked-diffusion gains, including 56.9% of LLaDA-8B oracle opportunity.
The paper studies when masked diffusion language models should change unmasking strategy during generation rather than keep a fixed inference policy. It organizes decisions into five axes and measures adaptation opportunity as the one-step gain of the best alternative over a validation-chosen action. Across three MDMs and ten tasks, those opportunities are highly uneven but sometimes concentrated and predictable. On LLaDA-8B constrained JSON filling, adapting only the top 10% of states captures 56.9% of the candidate-set oracle opportunity.
- Inference choices are grouped into score, cardinality, region, commitment, and planning.
- Adaptation value depends on how often and how strongly strategies reverse.
- Opportunities vary widely across three MDMs and ten tasks.
- Adapting the top 10% of states captures 56.9% of oracle opportunity on LLaDA-8B JSON filling.
From the original article
1 min read at huggingface.co
Masked diffusion language models (MDMs) admit flexible generation orders, making the unmasking strategy an inference decision. Existing methods vary in how they prioritize positions, control parallelism, restrict selection regions, revise predictions, or plan future denoising, yet it remains unclear when these choices should change during generation.
By Injin Kong, Sunghwan Choi, Yohan Jo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.