Factorized MLLM embeddings for composed image retrieval
Original title: Factorized MLLM Embeddings for Composed Image Retrieval
FactEmbed decouples compositional query comprehension from discriminative embeddings using factorized latents and contrastive losses for improved CIR performance.
FactEmbed decouples compositional query comprehension from building discriminative embeddings using factorized latents that simulate intended visual modifications. It optimizes a factor-level contrastive loss and distills representations into a deeper attention layer of the MLLM. For efficient inference, it retains the [EOS] token for global embedding and uses mean-pooled factorized latents to rerank candidates.
Unconfirmed: Performance claims based on authors' evaluations.
- Decouples query comprehension from embeddings
- Factor-level contrastive loss
- Distills decomposed representations
- Retains [EOS] token for global embedding
- Mean-pooled latents for reranking at negligible cost
- Organizations
- Hugging Face
- AI models
- FactEmbed
From the original article
1 min read at huggingface.co
Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM's compositional understanding remains underutilized.
By WonJun Moon, Seongchan Kim, Dahun Kim · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.