titleWhen Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision
authorsYupeng Tang, Mingfeng Lin
date2026-11-12
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.16694
affiliationGeorgia Institute of Technology
upvotes3 (Hugging Face Daily Papers, 2026-11-19)
linksabs pdf hf code
cite
relatedScience Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems
PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
From Knowledge Access to Source Learning: Developing Source-Specific Competence
Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer
abstractWhen language models revise a decision after another agent disagrees, neither reversal counts nor final accuracy shows whether they are fixing mistakes or abandoning correct answers. We introduce a controlled two-pass audit that holds the question, the evidence, and the model's initial answer fixed and varies only the revision context: neutral reflection, anonymous disagreement, or disagreement attributed to a domain expert. For binary decisions the accounting is exact: revision raises accuracy only if it is selective enough, reversing wrong answers more than a/(1-a) times as often as correct ones, where a is initial accuracy (nine times at 90%). Across five models and 1,989 evidence-grounded decisions in medicine, science, and law, no model clears this bar under expert disagreement, and accuracy falls by up to 16.1 points. The expert label, a prestige cue, adds reversals for four of the five models, yet its effect on accuracy ranges from a 5.2-point loss to a 2.4-point gain, so reversal counts are a poor proxy for harm. With identical arguments from both sources, the label still raises reversal for three proprietary models and in a pooled analysis of four newer models. Models also respond to stated source reliability. Revision pipelines should therefore be evaluated with paired, conditional metrics, per model and protocol.

bibtex

@misc{tang2026when,
  title={When Expert Disagreement Hurts: Prestige and Selectivity in LLM Decision Revision},
  author={Yupeng Tang and Mingfeng Lin},
  year={2026},
  eprint={2611.16694},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.16694}
}