How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining

Lin Chen; Bolin Ni; Qi Yang; Lan Jiang; Kun Ding; Xiaoran Fan; Hower Yang; Ying Wang; Shiming Xiang

title	How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
authors	Lin Chen, Bolin Ni, Qi Yang, Lan Jiang, Kun Ding, Xiaoran Fan, Hower Yang, Ying Wang, Shiming Xiang
date	2026-09-28
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2609.35457
affiliation	Tencent Hunyuan
upvotes	61 (Hugging Face Daily Papers, 2026-11-01)
links	abs pdf hf
abstract	Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instead learn visual representations directly from raw pixels, offering a simple and unified architecture, but their scaling behavior has not been systematically characterized. To fill this gap, we compare scaling laws for encoder-free and encoder-based MLLMs and report three main findings: (1) Removing the visual encoder shifts the compute-optimal allocation for the multimodal objective toward larger models, while leaving that for text nearly unchanged. (2) The two architectures exhibit nearly overlapping loss--compute frontiers on the text objective, but diverge on the multimodal objective: encoder-free models underperform at small scales yet are predicted to catch up at around 10^{22} FLOPs, well within practical pretraining budgets. (3) Without a visual encoder, the language model learns to take over its role via vision-specific adaptation: bidirectional interactions among visual tokens become increasingly beneficial as training compute grows, visual processing shifts toward earlier layers, and expert routing for visual tokens becomes more concentrated. Overall, our results indicate that the advantage of the visual prior provided by a pretrained encoder diminishes with scale, positioning encoder-free architectures as a promising direction for multimodal pretraining.

cite

@misc{chen2026how,
  title={How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining},
  author={Lin Chen and Bolin Ni and Qi Yang and Lan Jiang and Kun Ding and Xiaoran Fan and Hower Yang and Ying Wang and Shiming Xiang},
  year={2026},
  eprint={2609.35457},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2609.35457}
}