CulturalRiddles benchmarks LLMs on riddles from 62 cultures
Original title: CulturalRiddles: A Multicultural Benchmark of Riddles
Across 28 model configurations, even frontier systems reach at most 22% accuracy in some communities.
Researchers introduce CulturalRiddles, a benchmark of original, human-written riddles spanning 62 cultures, 45 countries, and 51 languages. The riddles require abductive and compositional reasoning plus cultural, factual, social, and language knowledge. The authors evaluate 28 LLM configurations with automatic scoring and large-scale human evaluation. They publicly release the artifacts and open a contribution process to expand linguistic and cultural coverage.
- Riddles were created entirely by humans through a participatory community effort.
- Coverage is 62 cultures, 45 countries, and 51 languages.
- Twenty-eight LLM configurations were benchmarked.
- Frontier models reach at most 22% accuracy in some communities.
From the original article
1 min read at huggingface.co
As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts.
By My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.