| title | ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty |
|---|---|
| authors | Yuyang Dai, Yuting Tang, Futoshi Futami, Takashi Ishida |
| date | 2026-10-12 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.17016 |
| affiliation | ishidalab |
| upvotes | 9 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves. |
cite
@misc{dai2026probabilitybench,
title={ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty},
author={Yuyang Dai and Yuting Tang and Futoshi Futami and Takashi Ishida},
year={2026},
eprint={2610.17016},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.17016}
}