titleAPT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
authorsKechun Xu, Zhenjie Zhu, Anzhe Chen, Rong Xiong, Yue Wang
date2026-06-10
venuearXiv (preprint, not peer reviewed)
idarXiv:2606.12366
affiliationZhejiang University
upvotes2 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
primerVision-language-action models
abstractVision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the π and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/

cite

@misc{xu2026apt,
  title={APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies},
  author={Kechun Xu and Zhenjie Zhu and Anzhe Chen and Rong Xiong and Yue Wang},
  year={2026},
  eprint={2606.12366},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.12366}
}