| title | MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models |
|---|---|
| authors | Wenjie Zhu, Yabin Zhang, Wenjun Zeng, Lei Zhang |
| date | 2026-08-01 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.27637 |
| affiliation | VCLab |
| upvotes | 4 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Multimodal Large Language Models (MLLMs) have achieved strong performance on a wide range of vision-language tasks, but often fail under imperfect or shifted contexts. A reliable MLLM should refuse truly out-of-context (OOC) questions with subject-level context shifts while still answering shifted in-context (Shifted IC) questions with non-subject context shifts. Existing benchmarks mainly target OOC or visually unanswerable questions, but overlook answerable Shifted IC cases and cover limited OOC shifts. To fill this gap, we present MMOOC, a large-scale benchmark for evaluating refusal and robust answering abilities of MLLMs. MMOOC contains over 41K image-question pairs, including answerable Shifted IC cases and unanswerable OOC cases, spanning three question formats, eight shift types and six visual scenarios, with data quality ensured through MLLM-based filtering and human verification. We evaluate model responses using Accuracy and Refusal Rate, and further introduce an LLM-as-a-Judge metric to assess the correctness of model reasoning. Experiments on diverse MLLMs show that current models still struggle to balance answer-ability and refusal under shifted contexts. We further analyze key failure patterns and show that post-training can improve robustness. MMOOC will be made publicly available. |
cite
@misc{zhu2026mmooc,
title={MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models},
author={Wenjie Zhu and Yabin Zhang and Wenjun Zeng and Lei Zhang},
year={2026},
eprint={2607.27637},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.27637}
}