| title | Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time |
|---|---|
| authors | Jeongeun Park, Juhan Park, Taekyung Kim, Sungjoon Choi, Dongyoon Han, Sangdoo Yun |
| date | 2026-06-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.15631 |
| affiliation | NAVER AI Lab |
| upvotes | 16 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| primer | Vision-language-action models |
| abstract | Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot. |
cite
@misc{park2026retrieve,
title={Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time},
author={Jeongeun Park and Juhan Park and Taekyung Kim and Sungjoon Choi and Dongyoon Han and Sangdoo Yun},
year={2026},
eprint={2606.15631},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.15631}
}