| title | When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision |
|---|---|
| authors | Yupeng Tang, Mingfeng Lin |
| date | 2026-11-12 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2611.16694 |
| affiliation | Georgia Institute of Technology |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-19) |
| links | abs pdf hf code |
| cite | |
| related | Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems PACT: Can Enterprise AI Assistants Be Trusted Under Pressure? UniSD: Towards a Unified Self-Distillation Framework for Large Language Models From Knowledge Access to Source Learning: Developing Source-Specific Competence Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer |
| abstract | When language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than a/(1-a) times as often as correct ones, where a is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol. |
bibtex
@misc{tang2026when,
title={When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision},
author={Yupeng Tang and Mingfeng Lin},
year={2026},
eprint={2611.16694},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2611.16694}
}