Retrieval heads stay sparse and shared across MLLM modalities
Original title: A Broader Look into MLLM's Retrieval Heads
Ranking training on those heads raises Qwen3-VL-8B page Recall@1 on MMDocIR by 6.03 points.
The paper tests whether multimodal retrieval heads remain a sparse, shared mechanism when audio-visual evidence is added. Paired inputs put the same answer in speech or on screen. Across 18 language, vision-language, and omni-modal checkpoints, omni-modal models largely keep their backbones' retrieval heads, and ablating them removes nearly all audio-visual retrieval. Heads shared across modalities are under 2% of all heads, yet ablating them also hurts document understanding, spatial perception, and visual mathematics. Training those heads with a ranking objective improves document retrieval across three model families.
- The study covers 18 language, vision-language, and omni-modal checkpoints.
- Shared cross-modal retrieval heads are fewer than 2% of all heads.
- Ablating the heads eliminates nearly all audio-visual retrieval.
- The ranking objective improves retrieval across three model families.
- Products
- MMDocIR
- AI models
- Qwen3-VL-8B
From the original article
1 min read at huggingface.co
Locating relevant evidence in long inputs is a core capability of Multimodal Large Language Models. Prior studies attribute this capability to retrieval heads, a sparse set of attention heads that transfer information from the context to the output. However, these analyses are confined to a limited set of modalities and the corresponding models.
By Jia Wang, Yuqing Li, Jiangnan Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.