dSpark distills unmasking schedules to speed diffusion language models
Original title: Learning When to Commit: Schedule Distillation for Fast Diffusion Language Models
On GSM8K the paper reports 10.12× and 5.12× speedups on LLaDA-8B-Instruct and Dream-7B-Instruct while keeping task quality.
Researchers introduce dSpark, a schedule-distillation method that decides when diffusion language models should commit tokens during parallel decoding. Unmasking schedules favor tokens whose reveal makes other positions easier to predict, then distill those schedules into the model with a KL-divergence regularizer. The paper evaluates mathematical reasoning and code generation and reports large GSM8K speedups over baselines on LLaDA-8B-Instruct and Dream-7B-Instruct with stronger task performance. Code is at the VILA-Lab dSpark repository.
- GSM8K speedup is 10.12× on LLaDA-8B-Instruct versus the paper’s baselines.
- The same GSM8K comparison is 5.12× on Dream-7B-Instruct.
- A KL-divergence regularizer is used to limit quality loss.
- Evaluation covers mathematical reasoning and code generation.
- Organizations
- VILA-Lab
- AI models
- LLaDA-8B-InstructDream-7B-Instruct
From the original article
1 min read at arxiv.orgDiffusion large language models (dLLMs) can generate multiple tokens in parallel, but often require many decoding rounds to complete a response. Efficient generation depends not only on which tokens are easy to predict, but also on how committing to them helps predict the remaining positions.
By Aidar Myrzakhan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.