titleStepAudio 3 Gen Technical Report
authorsBin Lin, Bo Zhao, Boyang Wang, Boyang Zhang, Boyong Wu, Chao Yan, Chen Geng, Chen Wu, Cheng Yi, Chengli Feng, Chenglin Zhu, DanNi Wan, Daxin Jiang, Dongqing Pang, Fei Tian, Feng Tian, Future Li, Gang Yu, Guanglong Yang, Jia Peng, Jiahao Song, Jiamin Fan, Jiangjie Zhen, Jianzheng Gao, Jun Chen, Li Xie, Lifang Zhang, Lingli Ji, Liying Shi, Lun Cai, Min Xu, Na Wang, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Ruijie Xiong, Runze Li, Shenghua Hu, Shi Qiu, Siqi Tu, Siyi Zhou, Tianjiao Deng, Wanying Lu, Weiming Niu, Wen Sun, WenWen Qu, Xiangyu Zhang, Xianwei Zhang, XiaoSu Su, Xing Chen, Xinyu Liu, Xuerui Yang, Yang Li, Yang Yang, Yechang Huang, Yibo Zhu, Yifan Zhang, Yiyang Xu, Yu Fu, Yu Luo, Yu Zhou, Yumang Wang, Yunzhou Ju, Yuxiang Yang, Zekai Liu, Zengwei Yao, Zhenwei Mou, Zheqi Dai, Zhiyue Wu, Zichao Zhou
date2026-09-11
venuearXiv (preprint, not peer reviewed)
idarXiv:2609.12945
upvotes100 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractWe introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.

cite

@misc{lin2026stepaudio,
  title={StepAudio 3 Gen Technical Report},
  author={Bin Lin and Bo Zhao and Boyang Wang and Boyang Zhang and Boyong Wu and Chao Yan and Chen Geng and Chen Wu and Cheng Yi and Chengli Feng and Chenglin Zhu and DanNi Wan and Daxin Jiang and Dongqing Pang and Fei Tian and Feng Tian and Future Li and Gang Yu and Guanglong Yang and Jia Peng and Jiahao Song and Jiamin Fan and Jiangjie Zhen and Jianzheng Gao and Jun Chen and Li Xie and Lifang Zhang and Lingli Ji and Liying Shi and Lun Cai and Min Xu and Na Wang and Peilin Li and Peng Yang and Pengfei Tan and Qingjian Lin and Ruijie Xiong and Runze Li and Shenghua Hu and Shi Qiu and Siqi Tu and Siyi Zhou and Tianjiao Deng and Wanying Lu and Weiming Niu and Wen Sun and WenWen Qu and Xiangyu Zhang and Xianwei Zhang and XiaoSu Su and Xing Chen and Xinyu Liu and Xuerui Yang and Yang Li and Yang Yang and Yechang Huang and Yibo Zhu and Yifan Zhang and Yiyang Xu and Yu Fu and Yu Luo and Yu Zhou and Yumang Wang and Yunzhou Ju and Yuxiang Yang and Zekai Liu and Zengwei Yao and Zhenwei Mou and Zheqi Dai and Zhiyue Wu and Zichao Zhou},
  year={2026},
  eprint={2609.12945},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2609.12945}
}