titleSpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
authorsTianhui Liu, Jie Feng, Zhiheng Zheng, Shengyuan Wang, Yiming Guo, Yanxin Xi, Hangyu Fan, Yong Li, Pan Hui
date2026-05-29
venuearXiv (preprint, not peer reviewed)
idarXiv:2605.31148
affiliationFIBLAB, Tsinghua University
upvotes3 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractHumans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such reasoning into actions in everyday 3D environments. Although recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks, it remains unclear whether they can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback. To study this problem, we introduce SpatialAct, a simulator-grounded benchmark for probing action-conditioned spatial reasoning in 3D scenes. Starting from the most challenging setting, Multi-turn Interactive Refinement, we further design its decomposed counterpart, Single-step Error Detection and Fix, together with five fundamental spatial ability tasks to diagnose the underlying causes of model failures. Experiments reveal a clear reasoning-to-action gap: current VLMs can perform well on isolated spatial reasoning tasks, but struggle to maintain coherent spatial beliefs and produce reliable actions during multi-turn feedback, substantially underperforming humans. These results suggest that current VLM agents still lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away.

cite

@misc{liu2026spatialact,
  title={SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes},
  author={Tianhui Liu and Jie Feng and Zhiheng Zheng and Shengyuan Wang and Yiming Guo and Yanxin Xi and Hangyu Fan and Yong Li and Pan Hui},
  year={2026},
  eprint={2605.31148},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.31148}
}