| title | StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models |
|---|---|
| authors | Zhe Liu, Jinghua Hou, Yuxiang Lu, Zhenya Yang, Xianzhe Fan, Junwei Luo, Junyi Li, Ruihua Han, Zhi Hou, Hengshuang Zhao |
| date | 2026-08-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.26067 |
| upvotes | 20 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| primer | Vision-language-action models |
| abstract | Vision-Language-Action (VLA) models have demonstrated effectiveness in robot manipulation, yet state-of-the-art models such as pi0.5 operate under a single-frame paradigm, limiting their ability to retain past observations and develop precise spatial perception. In this paper, we propose StreamPI, a streaming multimodal temporal modeling framework that equips single-frame VLA with temporal reasoning capability without introducing any additional parameters. One core design is instruction-anchored temporal modeling. It treats each (visual observation, language instruction) pair as an atomic temporal unit: bidirectional attention within each pair enables cross-modal fusion, while causal attention across pairs preserves autoregressive streaming inference. This ensures the language instruction serves as a persistent semantic anchor throughout task execution. To bridge the gap between synchronous training and asynchronous real-robot deployment, we introduce a andom-interval streaming training strategy: a proper inter-frame interval (e.g., every 3 frames) enables faster and smoother action execution. Beyond this, randomizing the interval further improves robustness to frame-timing perturbations, supporting asynchronous deployment in practice. Furthermore, by leveraging the length extrapolation capability of the LLM backbone, StreamPI seamlessly inherits pretrained single-frame weights and supports flexible single-frame and multi-frame inference. Experiments on real-robot tasks spanning memory-dependent and precise perception scenarios, as well as the simulation benchmark LIBERO, demonstrate that StreamPI outperforms pi0.5 across diverse tasks. |
cite
@misc{liu2026streampi,
title={StreamPI: Streaming Multimodal Temporal Modeling for Vision-Language-Action Models},
author={Zhe Liu and Jinghua Hou and Yuxiang Lu and Zhenya Yang and Xianzhe Fan and Junwei Luo and Junyi Li and Ruihua Han and Zhi Hou and Hengshuang Zhao},
year={2026},
eprint={2608.26067},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.26067}
}