Chaos in the Text: Revealing the Modality Preference in Mixed-Modality Retrievers
Mixed-modality retrievers favor text over images; Trident reduces that bias.
The paper finds dense retrievers stay strong on single-modality corpora but degrade sharply when text and image documents coexist, forming a V-shaped performance curve as images are replaced by equivalent text. Irrelevant text degrades retrieval more than the same number of irrelevant images because text representations receive systematically higher similarity scores, a bias the authors call Chaos in the Text. Their method, Trident, builds text, image, and fused views of each document as co-equal positives and trains with Multi-Positive View InfoNCE. On visual-document and natural-image benchmarks it improves mixed-modality retrieval for both CLIP-based and VLM-based retrievers and raises average single-modality performance.
- Mixed text-image corpora cause a V-shaped drop in retrieval quality.
- Irrelevant text hurts more than an equal number of irrelevant images.
- Text embeddings get higher similarity scores and outrank relevant images.
- Trident trains text, image, and fused views as co-equal positives.
Full article188 words · extracted from huggingface.co · click to collapse
Dense retrievers have made significant progress on text and image corpora, but whether these capabilities extend reliably to mixed corpora containing text, image, and fused text-image documents remains unclear. In this paper, we systematically examine retrievers across architectures and find that their performance is highly sensitive to modality composition. As image documents are progressively replaced with semantically corresponding text representations, retrieval performance follows a pronounced V-shaped curve, remaining strong on single-modality corpora but degrading substantially when modalities coexist. In particular, irrelevant text causes more severe degradation than an equal number of irrelevant images, a phenomenon we term Chaos in the Text. Further analysis reveals modality preference, whereby text representations receive systematically higher similarity scores, allowing irrelevant text to outrank relevant images. To mitigate this bias, we introduce Trident, which constructs text, image, and fused text-image views of each document as co-equal positives and jointly optimizes relevance discrimination and positive-view balance through Multi-Positive View InfoNCE. Experiments across visual document and natural image benchmarks show that trident improves mixed-modality retrieval on both CLIP-based and VLM-based architectures, reduces sensitivity to modality composition and text distractors, and increases average single-modality retrieval performance.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.11816