| title | JEV-as-a-Judge: Accept When Confident, Escalate When Unsure |
|---|---|
| authors | Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman |
| date | 2026-09-22 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.26550 |
| affiliation | Carnegie Mellon University |
| upvotes | 41 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost. |
cite
@misc{li2026jev,
title={JEV-as-a-Judge: Accept When Confident, Escalate When Unsure},
author={Yubo Li and Yidi Miao and Ramayya Krishnan and Rema Padman},
year={2026},
eprint={2609.26550},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.26550}
}