| title | Your Benchmark Is an Empirical Measure Over Difficulty |
|---|---|
| authors | Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang |
| date | 2026-10-21 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.29927 |
| affiliation | University of Illinois at Urbana-Champaign |
| upvotes | 18 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: measure mismatch, where the benchmark difficulty distribution differs from the intended difficulty distribution, and coverage gaps, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze 25 popular benchmarks spanning 7 domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor the model improvement process. Finally, we turn diagnosis into action by introducing BenchFill, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps. |
cite
@misc{sun2026your,
title={Your Benchmark Is an Empirical Measure Over Difficulty},
author={Yifan Sun and Naicheng Yu and Jiawen Gong and Jingyan Shen and Huan Zhang},
year={2026},
eprint={2610.29927},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.29927}
}