How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Scaling laws suggest encoder-free multimodal models catch encoder-based ones near 10^22 FLOPs as the visual prior matters less.
The paper compares scaling laws for encoder-free multimodal LLMs, which learn vision from raw pixels, with encoder-based models that use a pretrained visual encoder. Removing the encoder shifts compute-optimal allocation on the multimodal objective toward larger models, while the text objective is nearly unchanged. Loss-compute frontiers overlap on text, but encoder-free models lag at small scale and are predicted to catch up around 10^22 FLOPs. With more compute, the language model adapts through bidirectional visual-token interaction, earlier visual processing, and more concentrated expert routing.
- Encoder removal shifts multimodal compute-optimal allocation toward larger models.
- Text loss-compute frontiers nearly overlap for both architectures.
- Encoder-free models are predicted to catch up near 10^22 FLOPs.
- Visual processing moves earlier and expert routing grows more concentrated.
From the original article
1 min read at huggingface.co
Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized.
By Lin Chen, Bolin Ni, Qi Yang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.