| title | Your Embedding Model is SMARTer Than You Think |
|---|---|
| authors | Jianrui Zhang, Hyun Jung Lee, Sukanta Ganguly, Tae-Eui Kam, Donghyun Kim, Yong Jae Lee |
| date | 2026-05-24 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.24938 |
| upvotes | 23 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Multimodal retrieval relies heavily on single-vector retrievers, which compress rich, sequential token sequences into one single global representation. While efficient, they discard fine-grained, local evidence critical for dense retrieval tasks. Multi-vector approaches were introduced as a solution, but they strictly require training and many ignore the necessity of a globally summarizing representation. To address this, we introduce SMART, a framework that unlocks the latent multi-vector capabilities of standard single-vector models. We first demonstrate that standard contrastive training on the pooled embedding implicitly shapes the retrieval geometry of preceding hidden states via gradient flow. By applying direct late-interaction over these frozen hidden states during inference, SMART acts as a plug-and-play upgrade that consistently improves performance across diverse modalities, improving even the state-of-the-art models further on MMEB-V2. We also reveal SMART's superior performance, as simple lightweight post-training not only saves time and compute, but also brings forth further improvement on Visual Document retrieval, allowing a single-vector model to outperform SoTA multi-vector counterparts. Ultimately, SMART offers both a highly efficient inference enhancement and a powerful finetuning technique for multimodal retrieval. We open source our code and weights at https://github.com/HanSolo9682/SMART. |
cite
@misc{zhang2026your,
title={Your Embedding Model is SMARTer Than You Think},
author={Jianrui Zhang and Hyun Jung Lee and Sukanta Ganguly and Tae-Eui Kam and Donghyun Kim and Yong Jae Lee},
year={2026},
eprint={2605.24938},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.24938}
}