titleVisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents
authorsZhengXian Wu, Qianhao Nie, Zhuohong Chen, Junjie Gao, Yuyao Zhang, Yunyao Yu, Hangrui Xu, Cheng Cheng, Guiyu Ma, Jin Zhang, Xingzhong Xu, Kai Yang, Haoqian Wang
date2026-11-03
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.04894
affiliationAnt Group
upvotes21 (Hugging Face Daily Papers, 2026-11-08)
linksabs pdf hf code
cite
relatedSearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration
UI-Venus-2 Technical Report
SWE-Trail: History-Guided Trajectory Collection for Coding Agent Training
SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narratives
abstractMultimodal agents use web and image tools to answer questions beyond their internal knowledge. Some search tasks use images mainly to identify an initial entity, then rely on text retrieval. These tasks offer limited insight into how agents connect information across images, objects, and search steps. We therefore focus on visual context management. This involves organizing and using visual information and linking it to other findings during search. Visually intensive tasks require agents to combine object appearance, local details, or spatial relations with external information. We introduce VisSearch-Bench, with 204 human-written tasks across 7 main domains and 25 subdomains. It covers Set-wise Visual Search, Sequential Visual Search, Hybrid Wide-Deep Search, Competitive Visual Disambiguation, and Visual-Structure-Guided Search. Each task provides a reference answer, traceable evidence, and scoring rules. Accuracy measures complete task success, while Process Score measures the correctness and completeness of individual results and subquestion answers. Across 16 multimodal models, the best system fully solves only 26.47% of tasks, even with search tools. These results highlight the challenge of linking visual evidence with search results to produce complete answers.

bibtex

@misc{wu2026vissearch,
  title={VisSearch-Bench: Evaluating Long Horizon Visual Search in Multimodal Agents},
  author={ZhengXian Wu and Qianhao Nie and Zhuohong Chen and Junjie Gao and Yuyao Zhang and Yunyao Yu and Hangrui Xu and Cheng Cheng and Guiyu Ma and Jin Zhang and Xingzhong Xu and Kai Yang and Haoqian Wang},
  year={2026},
  eprint={2611.04894},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.04894}
}