| title | CogniRoute: Learning to Route Social Evidence in Omni-Modal Models |
|---|---|
| authors | Yifan Shen, Pei Tian, Xinzhuo Li, Bowen Fang, Shujun Xia, Bingxuan Li, Ana Jojic, Wenming Ye, Xu Cao, James Matthew Rehg, Ismini Lourentzou |
| date | 2026-06-18 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.20970 |
| affiliation | PLAN Lab @University of Illinois Urbana-Champaign |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Omni-modal models can ingest video, audio, and text, but unified access to multiple modalities does not guarantee that a model uses the right evidence. This gap is especially pronounced in social video question answering, where the answer may hinge on a gesture, vocal tone, temporal cue, or mismatch between what is said and what is visually expressed. We introduce CogniRoute, a schema-guided Mixture-of-Experts framework for social omni reasoning. CogniRoute uses a training-only cognitive schema that factorizes each example by cross-modal relation, reasoning demand, and temporal scope, and aligns global routing signatures with this structure during supervised fine-tuning. We further introduce route-aware reinforcement learning, which jointly optimizes token generation and expert allocation using rewards for answer correctness, modality-consistent reasoning, and cognitive temporal grounding. To support training and evaluation, we construct OmniSocialBench, a diagnostic social video QA resource with 118K structured training examples, grounded reasoning traces, schema labels, temporal evidence spans, and a manually verified evaluation split. CogniRoute achieves 59.38\% average accuracy on OmniSocialBench, improving over the strongest proprietary baseline by 15.33 percentage points and the strongest open-source omni baseline by 26.77 points, with the largest gains on questions requiring audio-visual coordination, conflict resolution, and temporally grounded social inference. |
cite
@misc{shen2026cogniroute,
title={CogniRoute: Learning to Route Social Evidence in Omni-Modal Models},
author={Yifan Shen and Pei Tian and Xinzhuo Li and Bowen Fang and Shujun Xia and Bingxuan Li and Ana Jojic and Wenming Ye and Xu Cao and James Matthew Rehg and Ismini Lourentzou},
year={2026},
eprint={2606.20970},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.20970}
}