Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

Tzu-Chieh Wei; Yi-Cheng Lin; Huang-Cheng Chou; Kuan-Yu Chen; Hsin-Yen Sung; Shrikanth Narayanan; Hung-yi Lee

title	Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach
authors	Tzu-Chieh Wei, Yi-Cheng Lin, Huang-Cheng Chou, Kuan-Yu Chen, Hsin-Yen Sung, Shrikanth Narayanan, Hung-yi Lee
date	2026-06-19
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2606.21215
links	abs pdf hf code
abstract	As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.

cite

@misc{wei2026speaker,
  title={Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach},
  author={Tzu-Chieh Wei and Yi-Cheng Lin and Huang-Cheng Chou and Kuan-Yu Chen and Hsin-Yen Sung and Shrikanth Narayanan and Hung-yi Lee},
  year={2026},
  eprint={2606.21215},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.21215}
}