titleCultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
authorsPinzhen Chen, Koel Dutta Chowdhury, Xiaoya Xu, David Tan, Doreen Osmelak, Ona de Gibert, Ariun-Erdene Tumurchuluun, Ashok Urlana, Fedor Sizov, Hale Sirin, Jesujoba Alabi, Karrar Talib Abed, Mateusz Klimaszewski, Nikolay Bogoychev, Niyati Bafna, Patricia Schmidtova, Preksha Manjunath Shanbhag, Sherrie Shen, Vilem Zouhar, Vivek Iyer, Yasser Hamidullah, Yusser Al Ghussin, Zheng Zhao
date2026-08-10
venuearXiv (preprint, not peer reviewed)
idarXiv:2608.09766
affiliationQueen's University Belfast
upvotes2 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractMultilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone to contamination over time and overlooks locale and cultural considerations. We therefore advocate for source-contrastive evaluation and instantiate it with Cultivar, a localised subset of FLORES, which enables locale-specific translation evaluation. When paired with unlocalised counterparts, performance discrepancy allows the probing of data contamination and localisation robustness. We benchmark 32 open-weight models and find that MT-specialised models are less robust, a few models potentially overfit FLORES, and models tend to translate US content better than that of other locales, regardless of language.

cite

@misc{chen2026cultivar,
  title={Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness},
  author={Pinzhen Chen and Koel Dutta Chowdhury and Xiaoya Xu and David Tan and Doreen Osmelak and Ona de Gibert and Ariun-Erdene Tumurchuluun and Ashok Urlana and Fedor Sizov and Hale Sirin and Jesujoba Alabi and Karrar Talib Abed and Mateusz Klimaszewski and Nikolay Bogoychev and Niyati Bafna and Patricia Schmidtova and Preksha Manjunath Shanbhag and Sherrie Shen and Vilem Zouhar and Vivek Iyer and Yasser Hamidullah and Yusser Al Ghussin and Zheng Zhao},
  year={2026},
  eprint={2608.09766},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2608.09766}
}