Simplex embeddings cut the multimodal modality gap by 97–99%
Original title: Closing the Modality Gap via Simplex-Constrained Representations
Softmax adapters and Total Variation similarity match or beat unconstrained retrieval on five benchmarks.
The paper proposes moving multimodal contrastive embeddings, as in models such as CLIP and CLAP, from the unit sphere to the probability simplex. Nonnegative unit-mass constraints are used as a shared coordinate system for both modalities. Lightweight softmax adapters and Total Variation similarity cut the centroid L2 gap by 97–99% on five retrieval benchmarks while matching or exceeding unconstrained retrieval.
- Embeddings move from the unit sphere to the probability simplex.
- Both manifolds share the same intrinsic dimension.
- The centroid L2 gap falls 97–99% on five benchmarks.
- Retrieval matches or exceeds standard unconstrained models.
From the original article
1 min read at arxiv.orgWhile multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent \emph{modality gap}. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere, $S^{d-1}$, to the probability simplex, $Δ^{d-1}$.
By Shubham Gupta · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.