| title | HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games |
|---|---|
| authors | Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai |
| date | 2026-10-22 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.31162 |
| affiliation | Tencent |
| upvotes | 58 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| primer | Vision-language-action models |
| abstract | Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players. |
cite
@misc{tang2026histeve,
title={HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games},
author={Zecong Tang and Zian Guan and Yuncheng Guo and Zhanqiu Zhang and Yiwen Guo and Bo Dai},
year={2026},
eprint={2610.31162},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.31162}
}