| title | RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning |
|---|---|
| authors | Howard Qian, Yiting Chen, Yunfei Xie, Kejia Ren, Podshara Chanrungmaneekul, Gaotian Wang, Bowen Wen, Chen Wei, Kaiyu Hang |
| date | 2026-09-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.03199 |
| affiliation | Rice RobotPI Lab |
| upvotes | 11 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Robot learning increasingly depends on broad and diverse demonstrations, yet collecting robot data remains expensive and poorly suited to covering the long tail of real-world tasks. To address this bottleneck, we introduce RoboTok, an internet-scale data engine that, given a query human manipulation video, retrieves manipulation-relevant human demonstrations from web videos for training dexterous robot policies. Specifically, we learn a latent motion space from 3D hand trajectories expressed in estimated actor-centered reference frames. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions, while remaining compact enough for efficient search and continual indexing over internet-scale video collections. We evaluate RoboTok against existing robot-data retrieval approaches on retrieval benchmarks and downstream robot policy performance. Our results show that RoboTok retrieves more relevant manipulation demonstrations and improves downstream task success, establishing hand-pose trajectory-aware retrieval as a way to make web video a scalable and continuously growing source of supervision for robot learning. |
cite
@misc{qian2026robotok,
title={RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning},
author={Howard Qian and Yiting Chen and Yunfei Xie and Kejia Ren and Podshara Chanrungmaneekul and Gaotian Wang and Bowen Wen and Chen Wei and Kaiyu Hang},
year={2026},
eprint={2609.03199},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.03199}
}