| title | SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations |
|---|---|
| authors | Taewon Yun, Hyeonseong Park, Jeonghwan Choi, Hayoon Park, Yeeun Choi, Hwanjun Song |
| date | 2026-06-04 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.05563 |
| affiliation | Data Intelligence System Lab |
| upvotes | 40 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions. |
cite
@misc{yun2026socrates,
title={SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations},
author={Taewon Yun and Hyeonseong Park and Jeonghwan Choi and Hayoon Park and Yeeun Choi and Hwanjun Song},
year={2026},
eprint={2606.05563},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.05563}
}