UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models

Ioannis Maniadis Metaxas; Adrian Bulat; Alberto Baldrati; Anestis Zaganidis; Yassine Ouali; Hyeonuk Kim; Georgios Tzimiropoulos

title	UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
authors	Ioannis Maniadis Metaxas, Adrian Bulat, Alberto Baldrati, Anestis Zaganidis, Yassine Ouali, Hyeonuk Kim, Georgios Tzimiropoulos
date	2026-07-25
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2607.23373
affiliation	Samsung Research
upvotes	5 (Hugging Face Daily Papers, 2026-11-01)
links	abs pdf hf
abstract	Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.

cite

@misc{metaxas2026ultravit,
  title={UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models},
  author={Ioannis Maniadis Metaxas and Adrian Bulat and Alberto Baldrati and Anestis Zaganidis and Yassine Ouali and Hyeonuk Kim and Georgios Tzimiropoulos},
  year={2026},
  eprint={2607.23373},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2607.23373}
}