titleRobusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City
authorsAdrian Cespedes, Marcelo Chincha, Dunant Cusipuma, Victor Flores-Benites, David Ortega, Arturo Deza
date2026-06-18
venuearXiv (preprint, not peer reviewed)
idarXiv:2606.20980
affiliationArtificio
upvotes4 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractAs Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: https://huggingface.co/datasets/Artificio/robusto-2

cite

@misc{cespedes2026robusto,
  title={Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City},
  author={Adrian Cespedes and Marcelo Chincha and Dunant Cusipuma and Victor Flores-Benites and David Ortega and Arturo Deza},
  year={2026},
  eprint={2606.20980},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.20980}
}