| title | EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming |
|---|---|
| authors | Yaru Niu, Binghong Chen, Changwei Yao, Shangtao Li, Zhenlong Fang, Revanth Krishna Senthilkumaran, Shuai Zhou, Yuemin Mao, Kalie Ching, Po-Ju Chao, Yudong Luo, Peiqi Yu, Heng Zhang, Hao Zhang, Bingqing Chen, Chen Qiu, H. Eric Tseng, Ye Yuan, Umar Iqbal, Changliu Liu, Jonathan Francis, Ding Zhao |
| date | 2026-11-12 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2611.16728 |
| affiliation | Carnegie Mellon University |
| upvotes | 1 (Hugging Face Daily Papers, 2026-11-15) |
| links | abs pdf hf code |
| cite | |
| related | Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation Training Object Permanence in World Models Into the Unknown: Benchmarking AI agents on open scientific research problems K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand |
| abstract | Generalist manipulation policies can benefit from the complementary experience of humans, real robots, and simulated agents. However, differences in morphology, sensing, and control interfaces make it difficult to learn effectively from their combined demonstrations. We introduce EgoX, an egocentric cross-embodiment learning framework that organizes heterogeneous demonstrations through semantically corresponding observation and action modalities. EgoX pretrains a Modularized Cross-Embodiment Transformer with Embodiment Dreaming (MXT+) on this mixture and fine-tunes it on target-embodiment demonstrations. Within each embodiment, MXT+ assigns a dedicated tokenizer to each observation modality and a dedicated expert to each action modality, with a Transformer trunk shared across embodiments. A training-only embodiment-dreaming objective predicts latent representations of future visual and proprioceptive states, encouraging the shared trunk to model each embodiment's dynamics. Across 13 task-embodiment pairs spanning five tasks and six robot embodiments, including five physical robots and one simulated humanoid, cross-embodiment pretraining raises MXT+'s average normalized task progress from 0.63 to 0.80 over training from scratch. The gains are particularly pronounced under out-of-distribution conditions, where average task progress increases from 0.53 to 0.77. Ablations demonstrate the benefits of broader pretraining mixtures, modular interfaces, and embodiment dreaming, with dreaming producing more similar latent geometry across embodiments. Project page: egox-dream.github.io. |
bibtex
@misc{niu2026egox,
title={EgoX: Egocentric Cross-Embodiment Manipulation with Embodiment Dreaming},
author={Yaru Niu and Binghong Chen and Changwei Yao and Shangtao Li and Zhenlong Fang and Revanth Krishna Senthilkumaran and Shuai Zhou and Yuemin Mao and Kalie Ching and Po-Ju Chao and Yudong Luo and Peiqi Yu and Heng Zhang and Hao Zhang and Bingqing Chen and Chen Qiu and H. Eric Tseng and Ye Yuan and Umar Iqbal and Changliu Liu and Jonathan Francis and Ding Zhao},
year={2026},
eprint={2611.16728},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2611.16728}
}