Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation
Next-Concept Modeling generates text coarse-to-fine in parallel, up to 31 times faster than diffusion language models.
Next-Concept Modeling is a two-stage coarse-to-fine text generator that encodes text with a multiscale residual VQ-VAE and predicts all codes at one hierarchy level in parallel. At matched parameter counts it reports better generative perplexity than the strongest diffusion language model baseline on LM1B while running 21 times faster. On OpenWebText it is competitive and 10 times faster; on XSum it matches the best DLM ROUGE-L at 31 times the speed. On Sudoku it outperforms autoregressive and diffusion baselines across difficulty levels with fewer function evaluations.
- A multiscale residual VQ-VAE encodes text into hierarchical discrete codes.
- A transformer predicts every code in one hierarchy level in parallel.
- On LM1B, NCM beats the strongest DLM baseline and runs 21 times faster.
- On XSum it matches ROUGE-L at 31 times the speed; Sudoku needs fewer evaluations.
From the original article
1 min read at huggingface.co
Diffusion language models (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inference becomes a dominant computational workload. However, DLMs degrade quickly with fewer sampling steps, process the full sequence at every step, and require many passes over the dataset to reach good quality.
By Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.