HiLo-RD improves low-budget masked diffusion language models
Original title: HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models
Self-distillation from large-window states raises GSM8K accuracy by 9.4 points and cuts mean readable length by about 25%.
HiLo-RD is a high-to-low budget representation distillation method for masked diffusion language models that must both choose content and decide when to commit tokens. The authors say limited budgets cause premature commitment of answer fragments that then misdirect later reasoning. HiLo-RD pairs large-window teacher states from the same model with cropped small-window views and matches hidden representations at committed positions, without an external teacher. On mathematics benchmarks it improves low-budget accuracy and shortens responses; gains also appear in code generation without code-domain training.
Unconfirmed: Benchmark gains are the authors' reported results; independent replication is not stated.
- On GSM8K, accuracy rises by 9.4 percentage points.
- Mean readable length falls by about 25%.
- Traininguses the same model as teacher, with no external model.
- Gains transfer to code generation without code-domain training.
From the original article
1 min read at arxiv.orgWe propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate and where to commit it. This creates two coupled challenges: predicting useful content before reasoning is complete, and deciding when to fix answer tokens that shape later predictions.
By Xiang Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.