titleFrom VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation
authorsTingwen Du, Haoyu Guo, Hanxue Zhang, Xuanxu Wang, Junting Dong, Chao Gao, Mingfei Han, Tao Lu, Mulin Yu, Chunhua Shen, Xiaojun Chang, Zhihui Li
date2026-10-21
venuearXiv (preprint, not peer reviewed)
idarXiv:2610.29560
upvotes22 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
primerVision-language-action models
abstractVision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.

cite

@misc{du2026from,
  title={From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation},
  author={Tingwen Du and Haoyu Guo and Hanxue Zhang and Xuanxu Wang and Junting Dong and Chao Gao and Mingfei Han and Tao Lu and Mulin Yu and Chunhua Shen and Xiaojun Chang and Zhihui Li},
  year={2026},
  eprint={2610.29560},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.29560}
}