Paired-residual removal improves multilingual LLM text embeddings
Original title: Steering beyond Language: Improving Multilingual Training-Free Text Embeddings via Paired-Residual Subspace Removal
The inference-time method was tested on 18 languages and English bitext mining against more than 200 languages.
Researchers propose Paired-Residual Subspace Removal, a training-free inference-time method for multilingual text embeddings from frozen LLMs. It builds paired residuals from parallel sentences, estimates layer-wise low-rank subspaces, and suppresses those components in last-token hidden states. The authors report consistent gains over existing training-free baselines in monolingual and cross-lingual evaluations.
Unconfirmed: The claim that this is the first systematic study is the authors' own.
- More than 200 monolingual evaluations cover 18 languages on MTEB and MMTEB.
- Cross-lingual bitext mining compares English with over 200 other languages.
- The method targets language-specific variation while keeping semantic content.
- Gains are reported across different LLMs, which are not named.
From the original article
1 min read at arxiv.orgWhile deriving text embeddings directly from frozen LLMs has emerged as a promising direction, existing methods focus almost exclusively on English, leaving multilingual settings largely unexplored. In this work, we present the first systematic study of multilingual training-free text embeddings in LLMs.
By Ailiang Lin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.