titleWhere Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis
authorsJulia Herbst, Simon Roschmann, Quentin Bouniot, Zeynep Akata
date2026-11-09
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.12869
affiliationTechnical University of Munich
upvotes6 (Hugging Face Daily Papers, 2026-11-14)
linksabs pdf hf code
cite
relatedEvery Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation
On the Fragility of Tasks under Composition: Implicit Budgeting in Language Models
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Trees to Flows and Back: Unifying Decision Trees and Diffusion Models
abstractThe Platonic Representation Hypothesis (PRH) suggests that neural networks are converging toward similar representation spaces. As language models become more capable, their representations are aligning with those of vision models. However, recent work has questioned whether this trend continues to hold for newer language models. Moreover, it has been shown that representation similarity metrics can be confounded by model depth and width, requiring appropriate baselines for interpretation. In this work, we first investigate where alignment emerges between unimodal models across model layers and representation dimensions. Next, we find that the PRH capability-alignment trend persists for newer models as long as the captions are sufficiently clean and informative. Finally, we contextualize cross-modal alignment, introducing unimodal and multimodal baselines on various datasets. Our results show that the cross-modal representational alignment of unimodal models can surpass that of explicitly aligned vision-language models, but remains substantially behind the alignment observed for models of the same modality. Code is available at https://github.com/ExplainableML/prh-closer-look.

bibtex

@misc{herbst2026where,
  title={Where Does Alignment Emerge and What Does It Mean? A Closer Look at the Platonic Representation Hypothesis},
  author={Julia Herbst and Simon Roschmann and Quentin Bouniot and Zeynep Akata},
  year={2026},
  eprint={2611.12869},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.12869}
}