| title | OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation |
|---|---|
| authors | Jun Zhan, Chen Yang, Yitian Gong, Donghua Yu, Kuangwei Chen, Wenbo Zhang, Kexin Huang, Qi Luo, Zhe Xu, Ying Zhu, Jin Wang, Tengyue Zhang, Qi Chen, Cheng Chang, Songlin Wang, Junqi Dai, Jiasheng Ye, Xiaogui Yang, Tianyi Liang, Xiangyu Peng, Zhaoye Fei, Shimin Li, Qinyuan Cheng, Xie Chen, Xinchi Chen, Xipeng Qiu |
| date | 2026-07-26 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.23855 |
| affiliation | OpenMOSS |
| upvotes | 27 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Recent generative models are moving beyond silent video or standalone audio synthesis toward the joint generation of synchronized audio and video. Despite this progress, jointly generating audio and video with fine-grained cross-modal correspondence remains challenging due to their fundamental structural differences. Most existing methods use audio and video VAEs trained separately. As a result, the two latent spaces lack cross-modal alignment, leaving the downstream generative model to learn cross-modal synchronization from scratch. We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations. Beyond reconstruction, OmniVAE uses a segment-level audio-video contrastive objective to capture temporal-semantic correspondence and align the two latent spaces. In parallel, it distills features from pretrained modality-specific semantic encoders into each modality, improving the downstream learnability of both latent spaces. Extensive experiments show that both objectives consistently improve the learnability of the latent spaces, translating into higher generation quality and more accurate cross-modal synchronization in downstream text-to-audio-video generation. These findings underscore the importance of learning unified representations as a foundation for omnimodal modeling.1 |
cite
@misc{zhan2026omnivae,
title={OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation},
author={Jun Zhan and Chen Yang and Yitian Gong and Donghua Yu and Kuangwei Chen and Wenbo Zhang and Kexin Huang and Qi Luo and Zhe Xu and Ying Zhu and Jin Wang and Tengyue Zhang and Qi Chen and Cheng Chang and Songlin Wang and Junqi Dai and Jiasheng Ye and Xiaogui Yang and Tianyi Liang and Xiangyu Peng and Zhaoye Fei and Shimin Li and Qinyuan Cheng and Xie Chen and Xinchi Chen and Xipeng Qiu},
year={2026},
eprint={2607.23855},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.23855}
}