| title | Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval |
|---|---|
| authors | Suhyeong Park, Junha Jung, Jungwoo Park, Jaewoo Kang |
| date | 2026-07-06 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.04605 |
| affiliation | Korea University |
| upvotes | 19 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into K representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With K=64, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by 16.09times, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select. |
cite
@misc{park2026do,
title={Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval},
author={Suhyeong Park and Junha Jung and Jungwoo Park and Jaewoo Kang},
year={2026},
eprint={2607.04605},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.04605}
}