| title | RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models |
|---|---|
| authors | Bin Yu, Yao Zhang, Haishan Liu, Shijie Lian, Yuliang Wei, Xiaopeng Lin, Zhaolong Shen, Changti Wu, Ruina Hu, Bailing Wang, Cong Huang, Kai Chen |
| date | 2026-06-01 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.02277 |
| affiliation | Zhongguancun Academy |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| primer | Vision-language-action models |
| abstract | Vision-language-action (VLA) models are built on the premise that semantic understanding from pretrained language or vision-language backbones should guide robot action prediction. Yet robot fine-tuning is optimized as imitation over task-specific action distributions, and many evaluations can be solved through visual or instruction-action shortcuts. We introduce RoboSemanticBench (RSB), an embodied benchmark for diagnosing semantic grounding in action prediction: whether post-trained VLA models can use complex instruction semantics to select and manipulate the correct physical target. In each episode, a robot receives a multiple-choice math or general-knowledge question, observes candidate answer blocks, and must grasp the block corresponding to the correct answer. RSB covers controlled arithmetic, grade-school mathematical understanding, and commonsense or factual understanding under four-choice and ten-choice suites. Across representative VLA models, we find that many policies learn to grasp candidate blocks but select the semantically correct block at near-random or below-random rates after controlling for grasp success, revealing a persistent gap between backbone-level semantic competence and action prediction. |
cite
@misc{yu2026robosemanticbench,
title={RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models},
author={Bin Yu and Yao Zhang and Haishan Liu and Shijie Lian and Yuliang Wei and Xiaopeng Lin and Zhaolong Shen and Changti Wu and Ruina Hu and Bailing Wang and Cong Huang and Kai Chen},
year={2026},
eprint={2606.02277},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.02277}
}