Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

Siting Li; Zhengyang Wang; Simon Shaolei Du; Xi Chen; Yang Liu

title

authors

Siting Li, Zhengyang Wang, Simon Shaolei Du, Xi Chen, Yang Liu

date

2026-09-08

venue

arXiv (preprint, not peer reviewed)

arXiv:2609.09143

affiliation

Amazon Frontier AI & Robotics

upvotes

10 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf code

abstract

Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.

@misc{li2026studying,
  title={Studying Image Tokenizers as Visual Languages in Unified Multimodal Models},
  author={Siting Li and Zhengyang Wang and Simon Shaolei Du and Xi Chen and Yang Liu},
  year={2026},
  eprint={2609.09143},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2609.09143}
}