titleWhere to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?
authorsLiyang Li, Muzhi Zhu, Zhiyue Zhao, Hengyu Zhao, Ke Liu, Linhao Zhong, Hao Chen, Chunhua Shen
date2026-05-31
venuearXiv (preprint, not peer reviewed)
idarXiv:2606.01247
affiliationZhejiang University
upvotes26 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractHumans can reproduce the viewpoint specified by a target image through active head and body motion, yet spatial intelligence in foundation models has largely been studied as passive understanding of pre-collected observations. We introduce Target Viewpoint Reproduction (TVR) -- an active task where an agent adjusts its viewpoint in a 3D environment until its observation matches a given target image -- and TVRBench, an indoor-simulation benchmark spanning scene scale and target-view visual richness. TVR is far from solved: on the evaluation split, the strongest open-source and closed-source models reach only 7.8% and 12.0% success. Fine-grained analysis identifies two consistent bottlenecks: off-the-shelf models struggle with multi-turn visual history, and performance drops sharply when viewpoint reproduction requires body translation rather than in-place rotation, exposing a gap in mapping spatial discrepancies to embodied movement. To study reducing this gap, we build a unified TVR post-training framework covering expert-trajectory SFT, rationale-supervised CoT-SFT, offline Single-turn GRPO, and on-policy Multi-turn GRPO from live simulator rollouts. Visual-action SFT supplies the main gain, raising a 9B open-source model to 50.8% success; Multi-turn GRPO provides targeted multi-room refinement and reaches 51.4% overall, while CoT supervision and Single-turn GRPO degrade closed-loop performance. These results establish TVRBench as a testbed for measuring and training foundation models that actively perceive and act in 3D environments. Our code, data, and models are available at https://github.com/aim-uofa/TVRBench.

cite

@misc{li2026where,
  title={Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?},
  author={Liyang Li and Muzhi Zhu and Zhiyue Zhao and Hengyu Zhao and Ke Liu and Linhao Zhong and Hao Chen and Chunhua Shen},
  year={2026},
  eprint={2606.01247},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.01247}
}