| title | Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents |
|---|---|
| authors | Yanjie Zhang, Nanchen Hu, Yushi Sun |
| date | 2026-09-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2609.32536 |
| upvotes | 7 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Audio language models can recognize spoken commands and invoke tools, but an agent must first decide whether the acoustic and conversational context warrants action. We introduce VGBench, a 1,018-item diagnostic benchmark for action-level addressedness across side-talk, self-talk, and speaker-switch scenarios. Each item uses a shared action space comprising silence, a tool call, and a natural-language answer. Speaker-switch pairs hold the specified words fixed while source, distance rendering, and a temporal boundary define a controlled wearer-to-bystander shift. Six raw Audio LLMs and three training-free adaptations often identify the target tool yet rarely withhold action under this shift; the highest raw switch mute rate is 14%. We then use VoxGate as a post-training case study. Supervised training mutes 91.3% of switched commands while choosing the correct tool for all nearby wearer commands and text-only controls. An exploratory GRPO stage has similar switch performance; side-talk accuracy rises from 68.4% to 70.9%, and self-talk muting from 52.0% to 60.0%. Factorized controls identify an independent source-change effect, while sensitivity to the far-field manipulation varies across acoustic renderings. The benchmark therefore measures multi-cue acoustic-context gating rather than isolated speaker identity. |
cite
@misc{zhang2026do,
title={Do Audio LLMs Listen Before They Act? Diagnosing Acoustic-Context Gating in Voice Agents},
author={Yanjie Zhang and Nanchen Hu and Yushi Sun},
year={2026},
eprint={2609.32536},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2609.32536}
}