titleReducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens
authorsSeongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok Lim
date2026-11-05
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.07298
affiliationNLP & AI - Korea University
upvotes6 (Hugging Face Daily Papers, 2026-11-08)
linksabs pdf hf
cite
relatedLAMAR: An Open Language-Aware Multilingual Alignment Reranker
RenderRank: Learning to Rerank Text with Compressed Visual Tokens
Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
MixLID: Token-Level Language Identification in Mixed-Language Text
SemBridge: Language Transfer in Sparse Encoders via Multilingual Semantic Bridges
abstractEmbedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.

bibtex

@misc{hong2026reducing,
  title={Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens},
  author={Seongtae Hong and Youngjoon Jang and Jungseob Lee and Chanjun Park and Heuiseok Lim},
  year={2026},
  eprint={2611.07298},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.07298}
}