titleGameRealm: Grounding Visual Learning in Interactive Games
authorsWeiguang Zhao, Yifan Wang, Ni Yang, Yating Wang, Xingyu Miao, Yankai Jiang, Mulin Yu, Tao Lu, Haoyu Guo, Junting Dong, Weinan Zhang, Chunhua Shen
date2026-11-17
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.23140
affiliationShanghai AI Laboratory
upvotes1 (Hugging Face Daily Papers, 2026-11-19)
linksabs pdf hf code
cite
relatedNCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
Accurate, Interdisciplinary and Transparent Structure-property Understanding with Deep Native Structural Reasoning
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Sage-Taiyi: A Metacognitive Architecture for Scientific Discovery
Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation
abstractVisual foundation models are central to physical intelligence, yet their development remains constrained by fragmented training datasets, inconsistent annotation quality, and limited multimodal supervision. To address these challenges, we introduce GameRealm, a large-scale, richly annotated dataset that grounds visual observations in synchronized, game-native supervision. GameRealm encompasses over 120 million RGB frames and 1,000 hours of gameplay footage across all views. Beyond its unprecedented scale, GameRealm provides rich multimodal supervision, including metric depth, camera parameters, synchronized audio, control inputs, and 3D point trajectories. Its diversity further extends to flexible view configurations, spanning single-view, stereo, multi-view, and panoramic recordings. To enable scalable acquisition across diverse game environments, we develop a unified data collection pipeline that combines human-operated interaction with agentic exploration. Extensive experiments across diverse visual learning tasks demonstrate that GameRealm consistently strengthens visual foundation models in both visual generation and geometric learning. These results establish GameRealm as a versatile and valuable data resource, unlocking new opportunities for advancing visual intelligence.

bibtex

@misc{zhao2026gamerealm,
  title={GameRealm: Grounding Visual Learning in Interactive Games},
  author={Weiguang Zhao and Yifan Wang and Ni Yang and Yating Wang and Xingyu Miao and Yankai Jiang and Mulin Yu and Tao Lu and Haoyu Guo and Junting Dong and Weinan Zhang and Chunhua Shen},
  year={2026},
  eprint={2611.23140},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.23140}
}