LLaCO aligns continuous text diffusion with token competition
Original title: LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling
An 8B LLaCO model uses spherical noising and semantic retention scheduling and is reported to rival comparable discrete diffusion models.
The paper says continuous diffusion for text misaligns geometric trajectories with time because embedding-norm decay makes Euclidean distance produce spurious token overtaking. It formalizes pairwise dynamics as Competition Threshold Theory and proposes Semantic Retention Scheduling so time follows identity decay. LLaCO adapts a pretrained backbone with an embedding directional adapter and timestep-conditioned modulation, using spherical noising as a geodesic interpolation toward a mask-biased endpoint. Instantiated as an 8B model, LLaCO is reported to beat prior continuous diffusion baselines and rival discrete diffusion models of comparable scale.
Unconfirmed: Outperformance and parity with discrete diffusion models are the authors' claims; scores and the backbone are not stated.
- High-dimensional overtaking is said to need extreme noise, compressing identity decay near the noisy endpoint.
- Spherical noising is intended to remove norm decay while keeping token competition.
- SRS reallocates the denoising trajectory in proportion to logarithmic rank decay.
- The abstract does not name the pretrained backbone or report numeric benchmark scores.
From the original article
1 min read at arxiv.orgContinuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression.
By Zijian Jia · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.