titleWhere to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
authorsYingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Lei Zhang, Jianmin Chen, Yuyang Xia, Botong Geng, Jinjian Wu, Qifeng Chen
date2026-11-04
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.06844
affiliationHKUST
upvotes2 (Hugging Face Daily Papers, 2026-11-11)
linksabs pdf hf
cite
relatedDeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment
The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments
Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants
CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
abstractA central yet under-explored question in Multimodal Large Language Models (MLLMs) is where to connect---how visual information from a hierarchical vision encoder should be wired into the layer-wise semantics of a large language model. Mainstream MLLMs inject a fixed single-layer visual representation into the LLM and treat all LLM layers as a single consumer, ignoring that different layers demand different visual granularities. Recent multi-layer fusion methods either compress hierarchical features on the encoder side or rely on predefined sparse or hierarchical layer-to-layer connections, and therefore stop short of modeling adaptive cross-layer interactions between the ViT and LLM hierarchies. We address this gap with DGP (Dynamic Gated Pathways), a routing mechanism that establishes all-to-all connections between every ViT layer and every LLM layer, with each LLM layer dynamically gating its preference over visual layers conditioned on its own semantic state. Built on top of LLaVA-1.5-7B and 13B, DGP delivers consistent gains over the LLaVA-1.5-7B baseline (e.g., +4.2 on SQA^{I}, +3.3 on MM-Vet, +3.1 on VizWiz and LLaVA^{W}) and surpasses prior multi-layer visual fusion methods on most benchmarks. Layer-wise routing analyses further show that different LLM layers do select different visual granularities on demand, supporting both the effectiveness and interpretability of the proposed dynamic gated pathways. The code is available: https://github.com/YingyingYan1/DGP.

bibtex

@misc{yan2026where,
  title={Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers},
  author={Yingying Yan and Jiaqi Tang and Wei Wei and Qianzhou Wang and Lei Zhang and Jianmin Chen and Yuyang Xia and Botong Geng and Jinjian Wu and Qifeng Chen},
  year={2026},
  eprint={2611.06844},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.06844}
}