Rethinking Projector Training in Multimodal LLMs
PORTAL initializes multimodal projectors from summary statistics and matches caption-trained alignment without paired data.
The paper argues that projector training in multimodal LLMs mainly performs coarse vision-language alignment. Using 10–20% of image-caption data recovers most gains, and projectors trained on different datasets can be linearly interpolated. PORTAL builds a training-free optimal-transport map between Gaussian feature approximations on a shared principal subspace. With no paired data, it matches a pipeline trained on 558k pairs across six LLM backbones, two vision encoders, and 16 benchmarks, outperforming it in 3 of 7 settings and reaching 98% on average elsewhere.
- Using 10–20% of projector data recovers most downstream gains.
- PORTAL needs only per-modality statistics, not paired captions or gradients.
- It matches 558k-pair training across six LLMs, two encoders, and 16 benchmarks.
- PORTAL wins in 3 of 7 settings and averages 98% elsewhere.
From the original article
1 min read at huggingface.co
Multimodal LLMs are typically trained by first aligning a vision encoder with a language model via projector training on large image-caption datasets, followed by joint training of the full multimodal model. Despite its widespread use, the role of the projector training stage remains poorly understood.
By Fabian Gröger, Naël Ouerghemi, Shuo Wen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.