Register Tokens for Bounded-State Reasoning in Diffusion Language Models
Register tokens let diffusion language models like LLaDA and Dream carry reasoning state across cleared chunks, gaining up to 19.5 points on code.
Researchers propose register tokens: dedicated fixed-position tokens whose continuous hidden states are trained to carry reasoning progress across generation chunks in masked diffusion language models. After decoding and clearing a chunk, the model continues from the prompt and the carried register state instead of retaining earlier text. On LLaDA and Dream, registers outperform discrete-text carry on every benchmark, with gains up to 8.5 points on math and 19.5 points on code. Registers are especially effective for bounded code generation and can be further refined with reinforcement learning on long-horizon reasoning tasks.
- Registers are dedicated fixed-position tokens trained to carry reasoning progress across generation chunks.
- Tested on LLaDA and Dream, registers outperform discrete-text carry on every benchmark.
- Gains reach 8.5 points on math and 19.5 points on code benchmarks.
- Registers are especially effective for bounded code generation spanning multiple chunks.
- Register-based reasoning can be refined with reinforcement learning on long-horizon tasks.
From the original article
1 min read at huggingface.co
Masked diffusion language models (dLLMs) generate text by iteratively denoising masked tokens with bidirectional attention. Extending reasoning across generation chunks normally requires keeping earlier generated text in context. We ask whether a dLLM can instead continue reasoning after that text is cleared, using only a fixed-size carried state.
By Albert Ge, Chandan Singh, Yufan Zhuang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.