| title | Linear Scaling Video VLMs for Long Video Understanding |
|---|---|
| authors | Cristobal Eyzaguirre, Jiajun Wu, Juan Carlos Niebles |
| date | 2026-05-29 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.31598 |
| affiliation | Stanford University |
| upvotes | 9 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Video vision-language models (VLMs) are increasingly used in long-horizon and streaming settings, yet most video encoders still rely on spatiotemporal self-attention, causing compute and latency to grow quadratically with the number of frames. Existing efficiency methods improve scalability but often lose accuracy relative to full self-attention, for example through aggressive frame/token dropping or coarse attention approximations. We introduce StateKV, an inference-time method that adapts pretrained long-video VLMs to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models spanning three families and multiple scales, StateKV remains close to full self-attention and consistently outperforms dominant sliding-window / recency-based streaming approximations, without fine-tuning or architectural changes. StateKV also reduces video-prefill cost measured FLOPs, enabling stronger accuracy at a fixed compute budget by running larger models. These results suggest a practical step toward scalable long-video understanding. |
cite
@misc{eyzaguirre2026linear,
title={Linear Scaling Video VLMs for Long Video Understanding},
author={Cristobal Eyzaguirre and Jiajun Wu and Juan Carlos Niebles},
year={2026},
eprint={2605.31598},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.31598}
}