CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
CanvasAnneal injects teacher reasoning traces into diffusion canvases during curriculum RL, improving diffusion LLMs on MATH500, Countdown, and Tau2.
CanvasAnneal is a curriculum-guided reinforcement learning framework for diffusion language models that addresses exploration bottlenecks in standard RL. It warm-starts exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas, then gradually removes this guidance so the model generates reasoning trajectories independently. Across mathematical reasoning and tool-use benchmarks, it improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and accelerates reward improvement, though gains are task-dependent.