LAW bases language modeling entirely on self-conditioning
Original title: Loopholing All the Way: Language Modeling with pure Self-Conditioning
After TinyGSM training, LAW solves 42.2% of GSM8K at 1024 NFE versus 34.5% for uniform loopholing discrete diffusion.
Researchers introduce LAW, a continuous generative model that uses only self-conditioning, or loopholing, by propagating the expected embedding under the denoiser and decoding tokens once at the end. On Sudoku it outperforms prior discrete and continuous diffusion models, including loopholing discrete diffusion. On Maze it is the most accurate from 2 to 64 NFE. Fine-tuning the pre-trained Duo model with LAW raises accuracy from 17.2% to 48.1%, and on OpenWebText LAW approaches autoregressive models at 64 NFE.
- GSM8K accuracy is 42.2% for LAW versus 34.5% for uniform loopholing discrete diffusion.
- Fine-tuning Duo with LAW lifts accuracy from 17.2% to 48.1%.
- LAW is the most accurate Maze model from 2 to 64 NFE.
- Code is published at github.com/ilievanadezhda/law.
From the original article
1 min read at arxiv.orgRecent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning?
By Adrien Deschenaux · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.