| title | Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions |
|---|---|
| authors | Jaewoo Ahn, Junseo Kim, Hyunseo Kim, Heeseung Yun, Jaehyeon Son, Zsolt Kira, Gunhee Kim |
| date | 2026-08-31 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.30428 |
| affiliation | Seoul National University |
| upvotes | 12 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Strategic deception by LLM and VLM agents has emerged as a central AI alignment and safety concern. Social-deduction games (where each player holds a hidden role and communicates with others to deduce identities) serve as the canonical testbed, particularly in multi-agent settings. Existing testbeds, however, are text-only and run on a single fixed agent configuration, missing the non-verbal sensorimotor channels treated as core by deception taxonomies and leaving it ambiguous whether an observed behavior reflects the underlying model or the surrounding harness. We introduce MineAmongUs, a 3D multimodal Among Us sandbox where imposter agents must deceive crewmates through joint verbal and non-verbal action. We also propose ARIA, a configurable VLM-agent harness that exposes five cognitive-component ablation axes; and an atom- and arc-level annotation scheme grounded in deception taxonomies and operationalized at scale by an LLM-as-a-Judge reaching near-human atom-labeling agreement. Empirical results show that VLM agents pursue imposter wins through joint verbal and non-verbal deception, with non-verbal channels emerging as the more decisive winning contributors across both harness ablation and cross-VLM evaluation. Taken together, our work opens a new path for embodied VLM-agent alignment research. |
cite
@misc{ahn2026lies,
title={Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions},
author={Jaewoo Ahn and Junseo Kim and Hyunseo Kim and Heeseung Yun and Jaehyeon Son and Zsolt Kira and Gunhee Kim},
year={2026},
eprint={2608.30428},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.30428}
}