titleRandom Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand
authorsGeorge Ling, Shanshan Zhong, Lijin Yang, Hao Yang, Nanxiang Li, Zhongzhan Huang
date2026-10-31
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.00883
affiliationCarnegie Mellon University
upvotes49 (Hugging Face Daily Papers, 2026-11-08)
linksabs pdf hf code
cite
relatedBenchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation
Training Object Permanence in World Models
Into the Unknown: Benchmarking AI agents on open scientific research problems
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
abstractBenchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results. To quantify estimation error, we derive an accurate analytical approximation to the error distribution when estimating full-benchmark scores from a subset, showing that most subsets yield small errors, while a few produce much larger errors, resulting in a long-tailed distribution. This means random sampling alone can achieve surprisingly strong performance with high probability, but remains unstable due to rare large errors. We further show that this instability is inherent to modern foundation models and persists even as more models are evaluated. Based on these findings, we introduce BestRand, a stable and low-cost benchmark compression method that preserves the efficiency of random sampling while substantially reducing its high-error tail. We validate BestRand on 70+ text, multimodal, and agent benchmarks, and further extend it to VLA evaluation. Compared with competitive methods, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, while requiring only 51.3% of the construction cost. It further reduces OOD estimation error by 21.8% on average across score-range, model-family, and temporal shifts. Together, our theory provides practical guidance for understanding and predicting the accuracy and reliability of benchmark compression, while BestRand enables stable and accurate compression at low cost. The code and data are publicly available onhttps://github.com/George-Ling3/BestRand.

bibtex

@misc{ling2026random,
  title={Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand},
  author={George Ling and Shanshan Zhong and Lijin Yang and Hao Yang and Nanxiang Li and Zhongzhan Huang},
  year={2026},
  eprint={2611.00883},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.00883}
}