| title | Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders |
|---|---|
| authors | Georgii Aparin, Vadim Popov, Tasnima Sadekova, Assel Yermekova |
| date | 2026-06-05 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.07473 |
| affiliation | HUAWEI Noah's Ark Lab |
| upvotes | 14 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Whisper, a widely adopted ASR model, is known to suffer from hallucinations - coherent transcriptions generated for non-speech audio entirely disconnected from the input. We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations. We extract audio encoder activations and evaluate two representation spaces: raw Whisper activations and Sparse AutoEncoder (SAE) latents. We show that both spaces encode linearly separable hallucination-related information, with discriminative power concentrated in a sparse feature subset and increasing toward deeper encoder layers. We propose two steering strategies: activation-space steering and SAE latent-space steering. SAE-based steering reduces hallucination rate from 72.63% to 14.11% for Whisper small and from 86.88% to 27.33% for Whisper large-v3 on the full non-speech test set, with small WER degradation on speech data, approaching the performance of fine-tuning-based methods. |
cite
@misc{aparin2026whisper,
title={Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders},
author={Georgii Aparin and Vadim Popov and Tasnima Sadekova and Assel Yermekova},
year={2026},
eprint={2606.07473},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.07473}
}