Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Multimodal Flow introduces MF-1, a fully continuous language-and-vision model competitive after 150B tokens.
Multimodal Flow is a fully continuous generative architecture that models text and images as ordered continuous hyperchunks with a shared chunk-causal flow-matching backbone. Instantiated as MF-1 and pretrained on 150B tokens at 0.6B, 1.2B, and 1.6B scales, it scores 82.8 on GenEval and DPG-Bench and 75.3 on VQAv2, MMBench, and POPE. Under matched data, optimization, and parameter budgets it beats representative hybrid and discrete unified models. Code and weights are released.
- Fully continuous flow model unifies language and vision without discrete visual tokens.
- MF-1 pretrained at 0.6B, 1.2B, and 1.6B with 150B tokens.
- Scores 82.8 on GenEval/DPG-Bench and 75.3 on VQAv2, MMBench, and POPE.
- Outperforms hybrid and discrete models under matched data and parameter budgets.
From the original article
1 min read at huggingface.co
We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck.
By Hongyuan Tao, Xinggang Wang, Lianghui Zhu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.