| title | VGI-Bench: Probing Visual Intelligence in Video Generation Models |
|---|---|
| authors | Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang, Zeyi Liu, Yuren Hao, Songcheng Cai, Keming Wu, Penghui Du, Kai Zou, Rui Yang, Chenkai Sun, Ke Yang, Ping Nie, Kelsey R Allen, Chenglong Wang, Michel Galley, Jianfeng Gao, ChengXiang Zhai |
| date | 2026-08-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.19583 |
| affiliation | University of Illinois at Urbana-Champaign |
| upvotes | 57 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/ |
cite
@misc{he2026vgi,
title={VGI-Bench: Probing Visual Intelligence in Video Generation Models},
author={Xuan He and Cong Wei and Yuhao Cheng and Linrui Ma and Yuxuan Zhang and Zuojun Li and Yuhao Wen and Jize Jiang and Zeyi Liu and Yuren Hao and Songcheng Cai and Keming Wu and Penghui Du and Kai Zou and Rui Yang and Chenkai Sun and Ke Yang and Ping Nie and Kelsey R Allen and Chenglong Wang and Michel Galley and Jianfeng Gao and ChengXiang Zhai},
year={2026},
eprint={2608.19583},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.19583}
}