SpecMRL beats full-size MRL with half the embedding size
Original title: SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum
On MTEB, SpecMRL outperforms standard MRL at every tested reduced dimension, with larger gains under more aggressive compression.
SpecMRL reorganizes Matryoshka Representation Learning with a parameter-derived spectral basis ordered by feature importance so prefix readout happens in that learned space. The method otherwise follows standard MRL. MTEB experiments show it exceeds MRL at all evaluated reduced dimensions. The authors report that half-dimensional SpecMRL embeddings surpass full-dimensional MRL.
- The spectral basis is meant to prioritize informative feature directions.
- Prefix readout is performed in the learned spectral space.
- Gains are larger when embeddings are compressed more aggressively.
From the original article
1 min read at arxiv.orgMatryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout.
By Zhongxin Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.