Frequency-aware remasking lifts diffusion language model test-time scaling
Original title: Test-Time Scaling for Diffusion Language Models with Frequency-Aware Remasking
MATH-500 gains reach 3.73 points for LLaDA-8B-Instruct and 7.33 points for Dream-7B-Instruct.
The paper treats diffusion language model test-time scaling as adaptive allocation of inference compute and decoding strategy. A fast base sampler solves easy questions; frequency-aware remasking then tracks how often each token prediction changes and commits frequently changing tokens on hard questions. On LLaDA-8B-Instruct and Dream-7B-Instruct, the method beats a strong adaptive baseline on MATH-500 and HumanEval. It also recovers some items still unsolved after 100 certainty-aware samples.
- HumanEval coverage gains reach 11.39 points for LLaDA and 10.98 for Dream.
- Remasking commits tokens whose predictions change often across denoising steps.
- On unsolved items, recovery ranges from 3.75% to 18.75% of examples.
- The base sampler handles easy questions before the second stage.
- AI models
- LLaDA-8B-InstructDream-7B-Instruct
From the original article
1 min read at huggingface.co
Diffusion language models (DLMs) generate text through iterative denoising, where decisions about which tokens to commit or remask strongly affect output quality. Existing certainty-aware strategies rely on instantaneous confidence or entropy, but these local signals can be misleading on difficult reasoning problems.
By Bowen Zuo, Yue Yu, Dongruo Zhou · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.