titleViT-Up: Faithful Feature Upsampling for Vision Transformers
authorsKrispin Wandel, Jingchuan Wang, Hesheng Wang
date2026-06-12
venuearXiv (preprint, not peer reviewed)
idarXiv:2606.14024
affiliationShanghai Jiao Tong University
upvotes9 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractVision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 PCK@0.10 on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 PCK@0.10, demonstrating that ViT-Up scales favorably with backbone capacity.

cite

@misc{wandel2026vit,
  title={ViT-Up: Faithful Feature Upsampling for Vision Transformers},
  author={Krispin Wandel and Jingchuan Wang and Hesheng Wang},
  year={2026},
  eprint={2606.14024},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.14024}
}