Trainable [DUMP] prefix tokens cut interference in retrieval embeddings
Original title: Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens
The method leaves architecture, pooling, and contrastive loss unchanged and reports gains on retrieval and long-context benchmarks.
A paper proposes prepending trainable [DUMP] tokens during embedding-model training so residual token-level signals, including attention-sink artifacts, weigh less in the final retrieval vector. Architecture, pooling, and the contrastive retrieval loss stay the same; only the prefix tokens are optimized. The authors report consistent gains over finetuned baselines for both encoder and decoder embedding models, and say [DUMP] tokens change activation and attention on low-information tokens.
Unconfirmed: Benchmark gains are the authors' reported results; independent replication is not stated.
- [DUMP] tokens are prepended and trained with the standard contrastive retrieval objective.
- Model architecture, pooling strategy, and training loss are left unchanged.
- Gains are reported across encoder and decoder embedding models on retrieval and long-context benchmarks.
- Analysis links the tokens to changed activation and attention on low-information tokens.
From the original article
1 min read at huggingface.co
Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training.
By Seongtae Hong, Youngjoon Jang, Jungseob Lee · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.