| title | DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents |
|---|---|
| authors | Tongsheng Ding, Zhen Luo, Yixuan Yang, Boyu Wang, Luyang Xie, Jinyu Yang, Feng Zheng |
| date | 2026-08-01 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.00486 |
| affiliation | Southern university of science and technology |
| upvotes | 11 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines. |
cite
@misc{ding2026dreamtraj,
title={DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents},
author={Tongsheng Ding and Zhen Luo and Yixuan Yang and Boyu Wang and Luyang Xie and Jinyu Yang and Feng Zheng},
year={2026},
eprint={2608.00486},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.00486}
}