| title | Video-Oasis: Rethinking Evaluation of Video Understanding |
|---|---|
| authors | Geuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Minho Shim, Yukyung Choi |
| date | 2026-07-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2603.29616 |
| affiliation | NAVER |
| upvotes | 64 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | The inherent complexity of video understanding makes it difficult to determine whether Video-LLM benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, shared criteria for evaluating video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the criteria for evaluating video understanding. In this work, we introduce Video-Oasis, a sustainable diagnostic suite for systematically auditing existing video understanding benchmarks. This audit reveals that 55\% of existing benchmark samples are solvable without visual input or temporal context. After filtering these shortcuts, the remaining video-native challenges expose a substantial capability gap: state-of-the-art models perform only marginally above random guessing. Building on these findings, we use the distilled challenges as a testbed to investigate which algorithmic design choices contribute to robust video understanding. We hope our work provides a practical foundation for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis. |
cite
@misc{lim2026video,
title={Video-Oasis: Rethinking Evaluation of Video Understanding},
author={Geuntaek Lim and Sungjune Park and Jaeyun Lee and Inwoong Lee and Taeoh Kim and Dongyoon Wee and Minho Shim and Yukyung Choi},
year={2026},
eprint={2603.29616},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2603.29616}
}