Lafa scales continuous flow language models to 8B
Original title: Large Flow Language Models
Trained on fewer than 30B tokens, its self-speculative decoding reaches 2.54 times autoregressive throughput at comparable quality.
Lafa is a blockwise flow language model with causal attention on the prefix and bidirectional asynchronous flow inside each block. The authors train 0.5B, 7B, and 8B versions with fewer than 30B tokens by adapting pretrained autoregressive models. On instruction following, reasoning, and coding, the learned flow matches or surpasses autoregressive quality and can be distilled into one- and few-step flow maps. A single-model scheme verifies flow drafts autoregressively.
Unconfirmed: The paper does not state whether Lafa weights are publicly released.
- Prefix caching and blockwise parallel decoding speed inference.
- Flow training preserves or improves pretrained autoregressive ability via multi-token prediction.
- Reported throughput is 2.54 times an autoregressive model at comparable quality.
- The comparison also beats multi-token prediction and discrete diffusion methods.
From the original article
1 min read at huggingface.co
Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established. We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner.
By Arnav Chavan, Chanhyuk Lee, Udbhav Bamba · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.