Learning Multimodal Embeddings with Evidence-Aligned Readout
EviAlign reads multimodal evidence at semantic boundaries, reaching 76.9 Recall@1 across 12 MMEB tasks.
EviAlign couples semantic evidence generation with boundary readout inside a shared multimodal large language model. Evidence is split into five semantic units; contextualized states at unit boundaries are aggregated into one normalized embedding and trained with generation plus contrastive retrieval. A 2-by-3 study finds the gain from consistent versus permuted organization rises from 0.65 points at length-based positions to 2.39 points at evidence boundaries. Across 12 MMEB retrieval tasks, EviAlign reaches 76.9 average Recall@1 using 500K training pairs while keeping single-vector indexing.
- EviAlign organizes evidence into five semantic units and reads states at each boundary.
- Generation and contrastive retrieval jointly train the shared multimodal model.
- Consistent organization beats permuted evidence by 2.39 points at evidence boundaries.
- It scores 76.9 average Recall@1 across 12 MMEB tasks with 500K pairs.
- Single-vector indexing and scoring are retained.
From the original article
1 min read at huggingface.co
Multimodal large language models can expose task-relevant evidence through generation, but producing useful evidence does not by itself determine how it enters a retrieval embedding. We study whether the semantic organization of that evidence can also specify where representations are read.
By Zirong Chen, Fuda Ye, Enjun Du · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.