titleCulturalRiddles: A Multicultural Benchmark of Riddles
authorsMy Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, Ayush Munot, Aylin Naebzadeh, Anastasiia Nikitina, Dina Pisarevskaya, Divas Poudel, Jebish Purbey, Md Rabius Sany Apu, Ahmad Mufli Ramadhan, Ralitza Nikolova Rizova, Diogo M. Serrano-Miranda, Vivek Gopalkrishna Silimkhan, Luis Fernando Solis Navarro, Kaiser Sun, Arya Suneesh, Manaswitha Sunkara, Tejaswi Sunkara, Jonathan E Swindell, Hanna Yukhymenko, Jingyi Zheng, Esther Adenuga, Ilia Afanasev, Wafa Aissa, Lisa Alazraki, Lauren C Altomare, Priscilla Amondi Amuok, Emile Anand, Gabriel Assis, José Matías Avalos, Silvia Lorena Avila, Zakaria Baannou, Niyati Bafna, Yash Bagla, Sarah Bahrami, Bontu F Balcha, Vanya Bannihatti Kumar, Irene Baucells, Abdumutalib Begmuratov, Naman Bhatia, Souvik Bhattacharyya, Isabel Blanco Ferreiro, Felipe Bochini, Sayuru Bopitiya, Angana Borah, Tulika Bose, Olena Bugaiova, Juliana Carvalho, Nancy F Chen, Stephy Chen, Rochelle Choenni, Jayanth Krishna Chundru, Leandro A Cossio, Nebojša Damnjanović, Ona de Gibert, Marco De Santis, Pedro Delgado, Kaustubh Dholé, Adnan El Assadi, Wesam El-Sayed, Nadine M Elframawy, Oumayma Essarhi, Naome A Etori, Reuben Chagas Fernandes, Saanidhi Pravin Gade, Javier García Gilabert, Álvaro González-Escapa, Bin Hu, Kaiyu Huang, Abderrahmane Issam, Danka Jokić, Ulugbek Khalilov, Dominik Krzemiński, Roy Ka-Wei Lee, Haichang Li, Chen Cecilia Liu, Zhengyuan Liu, Cedric Lothritz, Houman Mehrafarin, Antonio Miranda-Escalada, Fengran Mo, Mehrnaz Mofakhami, Wafaa Mohammed, Felipe Moreno-Vera, Prahitha Movva, Jannatul Nayem, Van Ngo, Joy Olusanya, Marc Pàmies, Avinash Kuney Pandey, Yurii Paniv, Fred Philippy, Vladislav Poritski, Dikshya Poudel, Joanna Radoła, Sara Rajaee, Swati Rajwal, Prerana Rane, Muhammad Ravi Shulthan Habibi, Lucas Resck, Shubhashis Roy Dipta, Ananya Sahu, Eduardo Sánchez, Borja Sánchez Clemente, Aleix Sant, Nobin Sarwar, Istiak Shihab, Santhoshini Pranathi Singaraju, Jane Smith, David Stap, Emily Z Su, Ananth K Suresh, Sajag Swami, Ruiyang Tan, Nazia Tasnim, Khalid Tounoussi, Patteera Triamamornwooth, Muhammad Saad Umar, Shankar Venkitachalam, Oksana Volchek, Lingbo Wang, Debela Desalegn Yadeta, Rushikesh Zawar, Jinghan Zhang, Mike Zhang, Weihua Zheng, Marzieh Fadaee, Akhil Arora, Károly Boczka, Julia Kreutzer
date2026-10-21
venuearXiv (preprint, not peer reviewed)
idarXiv:2610.29968
affiliationCohere Labs Community
upvotes40 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractAs LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.

cite

@misc{nguyen2026culturalriddles,
  title={CulturalRiddles: A Multicultural Benchmark of Riddles},
  author={My Chiffon Nguyen and Tianyi Hu and Vinod Anbalagan and Henry Gagnier and Pouya Sadeghi and Farah Abdou and Tidor-Vlad Pricope and Diego MMS Oliveira and Jocelyn K D'Arcy and Madeline Smith and Oluwaseun A Ajayi and Yusser Al Ghussin and Zena Al-Khalili and Micol Altomare and Shayan Amir and Ahanaf Aziz and Bryan Chen and Ruchira Dhar and Abdellah El Mekki and Rawan Mohammed Elframawy and Akhil Eppa and Silvia Fernandez and Alex Gimei and Arya Hariharan and Christian Hoang and RamMohanRao Kadiyala and Diana Kozachek and Peerat Limkonchotiwat and Ankita Maity and Di Mo and Ayush Munot and Aylin Naebzadeh and Anastasiia Nikitina and Dina Pisarevskaya and Divas Poudel and Jebish Purbey and Md Rabius Sany Apu and Ahmad Mufli Ramadhan and Ralitza Nikolova Rizova and Diogo M. Serrano-Miranda and Vivek Gopalkrishna Silimkhan and Luis Fernando Solis Navarro and Kaiser Sun and Arya Suneesh and Manaswitha Sunkara and Tejaswi Sunkara and Jonathan E Swindell and Hanna Yukhymenko and Jingyi Zheng and Esther Adenuga and Ilia Afanasev and Wafa Aissa and Lisa Alazraki and Lauren C Altomare and Priscilla Amondi Amuok and Emile Anand and Gabriel Assis and José Matías Avalos and Silvia Lorena Avila and Zakaria Baannou and Niyati Bafna and Yash Bagla and Sarah Bahrami and Bontu F Balcha and Vanya Bannihatti Kumar and Irene Baucells and Abdumutalib Begmuratov and Naman Bhatia and Souvik Bhattacharyya and Isabel Blanco Ferreiro and Felipe Bochini and Sayuru Bopitiya and Angana Borah and Tulika Bose and Olena Bugaiova and Juliana Carvalho and Nancy F Chen and Stephy Chen and Rochelle Choenni and Jayanth Krishna Chundru and Leandro A Cossio and Nebojša Damnjanović and Ona de Gibert and Marco De Santis and Pedro Delgado and Kaustubh Dholé and Adnan El Assadi and Wesam El-Sayed and Nadine M Elframawy and Oumayma Essarhi and Naome A Etori and Reuben Chagas Fernandes and Saanidhi Pravin Gade and Javier García Gilabert and Álvaro González-Escapa and Bin Hu and Kaiyu Huang and Abderrahmane Issam and Danka Jokić and Ulugbek Khalilov and Dominik Krzemiński and Roy Ka-Wei Lee and Haichang Li and Chen Cecilia Liu and Zhengyuan Liu and Cedric Lothritz and Houman Mehrafarin and Antonio Miranda-Escalada and Fengran Mo and Mehrnaz Mofakhami and Wafaa Mohammed and Felipe Moreno-Vera and Prahitha Movva and Jannatul Nayem and Van Ngo and Joy Olusanya and Marc Pàmies and Avinash Kuney Pandey and Yurii Paniv and Fred Philippy and Vladislav Poritski and Dikshya Poudel and Joanna Radoła and Sara Rajaee and Swati Rajwal and Prerana Rane and Muhammad Ravi Shulthan Habibi and Lucas Resck and Shubhashis Roy Dipta and Ananya Sahu and Eduardo Sánchez and Borja Sánchez Clemente and Aleix Sant and Nobin Sarwar and Istiak Shihab and Santhoshini Pranathi Singaraju and Jane Smith and David Stap and Emily Z Su and Ananth K Suresh and Sajag Swami and Ruiyang Tan and Nazia Tasnim and Khalid Tounoussi and Patteera Triamamornwooth and Muhammad Saad Umar and Shankar Venkitachalam and Oksana Volchek and Lingbo Wang and Debela Desalegn Yadeta and Rushikesh Zawar and Jinghan Zhang and Mike Zhang and Weihua Zheng and Marzieh Fadaee and Akhil Arora and Károly Boczka and Julia Kreutzer},
  year={2026},
  eprint={2610.29968},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.29968}
}