| title | UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos |
|---|---|
| authors | Yuxuan Zhang, Haozhong Xiong, Jiayi Song, Jinpeng Yu, Yang Shi, Jiaming Liu, Ruihua Huang, Liwei Wang |
| date | 2026-08-13 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.11752 |
| affiliation | Qwen Business Unit |
| upvotes | 14 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Talking-video character replacement requires coordinated transfer of appearance and voice while preserving the source motion, scene, linguistic content, and audio-video timing. Existing methods use separately optimized models for the two modalities, making audio-visual consistency difficult to enforce. We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos. Given a source video, a reference image, and a reference voice clip, UniSwap transfers the reference appearance and vocal timbre within a single audio-visual diffusion transformer while preserving the source content and dynamics. To address the scarcity of aligned cross-identity training pairs, we introduce a swap-and-reconstruct pipeline that removes visual and vocal identity from real clips and uses the original clips as reconstruction targets. Starting from a bidirectional backbone, we progressively adapt the model through In-context Pretraining for joint replacement, Conditional Streaming Adaptation for block-causal KV-cached generation, and Efficient Self-forcing DMD for mitigating exposure bias and reducing sampling from 30 to 3 denoising steps per block. Efficient Multi-LoRA Switching enables the three DMD roles to share a single frozen backbone. Feature-RoPE Decomposition keeps cached positions within the training range, supporting stable long-form inference. Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation. |
cite
@misc{zhang2026uniswap,
title={UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos},
author={Yuxuan Zhang and Haozhong Xiong and Jiayi Song and Jinpeng Yu and Yang Shi and Jiaming Liu and Ruihua Huang and Liwei Wang},
year={2026},
eprint={2608.11752},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.11752}
}