QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge
QuranicMMLU offers 980 questions testing generative models on Quranic Arabic across five linguistic pillars.
QuranicMMLU evaluates generative models on Quranic Arabic using a five-pillar taxonomy of phonology, morphology, syntax, semantics, and pragmatics, with 31 detailed phenomena. The set contains 980 human-reviewed questions, each in open-ended and multiple-choice form, stratified by Bloom cognitive level and verse perplexity. Across 12 systems, an Islamic-specialized model leads, but average multiple-choice accuracy is 84% versus 60% open-ended quality. Rankings agree closely (Kendall tau 0.73), while multiple choice conceals failures visible only without answer options.
- 980 human-reviewed questions appear in open-ended and multiple-choice forms.
- Five pillars cover phonology, morphology, syntax, semantics, and pragmatics across 31 leaves.
- Twelve systems average 84% multiple-choice accuracy versus 60% open-ended quality.
- An Islamic-specialized model leads; rankings agree with Kendall tau 0.73.
From the original article
1 min read at arxiv.orgWe introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quranic benchmarks center on general question answering and semantic retrieval, without probing specific linguistic competencies or stratifying by cognitive demand and verse difficulty.
By Rawan El Ghali · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.