| title | PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception |
|---|---|
| authors | Yana Wei, Hongbo Peng, Yanlin Lai, Liang Zhao, Kangheng Lin, En Yu, Keyu Lv, Han Zhou, Yin Tang, Haodong Li, Mitt Huang, Hangyu Guo, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Vishal M. Patel |
| date | 2026-06-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.28322 |
| affiliation | Johns Hopkins University |
| upvotes | 41 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation. |
cite
@misc{wei2026perceptionrubrics,
title={PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception},
author={Yana Wei and Hongbo Peng and Yanlin Lai and Liang Zhao and Kangheng Lin and En Yu and Keyu Lv and Han Zhou and Yin Tang and Haodong Li and Mitt Huang and Hangyu Guo and Jianjian Sun and Zheng Ge and Xiangyu Zhang and Daxin Jiang and Vishal M. Patel},
year={2026},
eprint={2606.28322},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.28322}
}