titleFATE: Frame-Level Audio-Visual Temporal Embedding
authorsKaisi Guan, Bingzi Zhang, Xihua Wang, Ying Ba, Xin Cheng, Yijing Chen, Ruihua Song
date2026-08-02
venuearXiv (preprint, not peer reviewed)
idarXiv:2608.01310
affiliationRenmin University of China
upvotes4 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractWhen a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at https://github.com/guankaisi/FATE.

cite

@misc{guan2026fate,
  title={FATE: Frame-Level Audio-Visual Temporal Embedding},
  author={Kaisi Guan and Bingzi Zhang and Xihua Wang and Ying Ba and Xin Cheng and Yijing Chen and Ruihua Song},
  year={2026},
  eprint={2608.01310},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2608.01310}
}