Discrete Beckmann Transport Models for One-Step Language Modeling and Reasoning
DBTM achieves one-step text generation via a time-independent transport map trained directly from data, removing pretrained teacher distillation.
Discrete Beckmann Transport Models (DBTM) build a time-independent flow whose autonomous transport map provably carries any point in ambient space to a fixed point on simplex vertices in a single step. The fixed-point property is characterized by a conservation equation whose residual can be minimized directly from data, eliminating the need for a teacher flow, distillation, and time conditioning. A partial-context interpolant extension turns additional function evaluations into refinement steps rather than ODE integration steps. On language modeling and reasoning tasks, DBTM's one- and few-step generation improves quality and accuracy over discrete diffusion and continuous flow baselines.
- Autonomous transport map reaches simplex vertex fixed point in a single step
- Trained by minimizing conservation-equation residual directly; no teacher flow or distillation required
- Partial-context interpolant makes extra evaluations refinement steps, not ODE steps
- Outperforms discrete diffusion and continuous flow baselines in one- and few-step generation
From the original article
1 min read at arxiv.orgDiscrete diffusion and flow models are a promising alternative to autoregressive language models, but compressing many-step sampling into fewer steps typically requires distilling a pretrained teacher model. This caps the student at the teacher's quality and requires a costly two-stage training pipeline.
By Sophia Tang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.