DAR-MGSM benchmark tests math reasoning in Arabic dialects
Original title: Benchmarking Arabic Dialects for Math Problems
Eleven models generally scored worse on culturally localized problems than on translated-only versions.
Researchers introduce DAR-MGSM, a manually curated benchmark of multi-step math word problems in five major Arabic dialects and Modern Standard Arabic. It includes parallel translated-only and culturally localized versions. Eleven Arabic-centric and multilingual models were evaluated, and performance generally declined on the localized set, with sensitivity varying by model. Translated and localized variants often produced different internal reasoning trajectories even when accuracy was similar.
- DAR-MGSM is presented as the first manual Dialectal Arabic math benchmark.
- Localized items adapt entities and references to regional varieties.
- Arabic-centric models showed different sensitivity to localization.
- Similar accuracy can still hide distinct internal reasoning paths.
From the original article
1 min read at arxiv.orgMathematical reasoning remains underexplored for low-resource languages, including Dialectal Arabic. To bridge this gap, we introduce the DAR-MGSM (Dialectal Arabic Multilingual Grade School Math) Benchmark, the first manually curated benchmark for evaluating mathematical reasoning in Dialectal Arabic.
By Noor Abo Mokh · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.