CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
CanvasAnneal injects teacher reasoning traces into diffusion canvases during curriculum RL, improving diffusion LLMs on MATH500, Countdown, and Tau2.
CanvasAnneal is a curriculum-guided reinforcement learning framework for diffusion language models that addresses exploration bottlenecks in standard RL. It warm-starts exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas, then gradually removes this guidance so the model generates reasoning trajectories independently. Across mathematical reasoning and tool-use benchmarks, it improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and accelerates reward improvement, though gains are task-dependent.
- Warm-starts RL exploration with teacher reasoning traces in the diffusion canvas.
- Curriculum gradually removes guidance for independent reasoning generation.
- Beats diffu-GRPO on MATH500, Countdown, and Tau2; gains task-dependent.
From the original article
1 min read at arxiv.orgDiffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration.
By Blake Olson · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.