Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Wendi Sang; Kai Li; Yifan Li; Jianqiang Huang; Xiaolin Hu

title	Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
authors	Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu
date	2026-10-23
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2610.32749
affiliation	Tsinghua Machine Learning Group
upvotes	123 (Hugging Face Daily Papers, 2026-11-01)
links	abs pdf hf code
dispatch	Leaner streaming model picks one voice out of crowded, noisy scenes
abstract	Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.

cite

@misc{sang2026efficient,
  title={Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes},
  author={Wendi Sang and Kai Li and Yifan Li and Jianqiang Huang and Xiaolin Hu},
  year={2026},
  eprint={2610.32749},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.32749}
}