Exponential residual flows give an ODE sampler for discrete diffusion
Original title: An ODE-based Sampler for Discrete Diffusion Models
The construction preserves the original diffusion's marginals and is specialized to masked and uniform diffusion.
The paper asks whether discrete diffusion models have a deterministic probability-flow analogue and introduces exponential residual flows. Each token and position gets a continuous residual that follows piecewise ODEs depending on the discrete state and the forward and backward rate matrices; the discrete state changes when a residual hits zero. The authors prove the construction preserves the original marginals, derive a practical discretization, and specialize it to masked and uniform diffusion. They report gains on code and text generation and say the trajectories support consistency distillation.
Unconfirmed: Reported generation gains are the authors' benchmark results.
- Residuals evolve deterministically until one reaches zero.
- Vector fields use the prescribed model's forward and backward rate matrices.
- Authors report improvements over prior samplers on code and text tasks.
- Deterministic trajectories are intended to support consistency distillation.
From the original article
1 min read at arxiv.orgThe marginals of a continuous diffusion model can be reproduced by a deterministic ordinary differential equation (ODE), known as the probability flow ODE, which provides a foundation for accelerated sampling and consistency distillation.
By Samson Gourevitch · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.