VākQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering
VakQA debuts a 2,001-pair Telugu spoken factoid QA benchmark, validating LLM-judge evaluation and benchmarking speech-input model reliability.
Researchers present VākQA, the first spoken question answering benchmark for Telugu, comprising 2,001 factoid question-answer pairs across six domains with 2.53 hours of speech audio, bilingual transcriptions, and human-verified references. They validate evaluation methods against human judgments, finding Gemini-as-a-judge closest to human ratings but non-uniformly strict, while open-weight judges penalize correct Telugu answers differing in surface form. Benchmarking proprietary and open-weight models shows translation loses cultural specificity, speech input causes phonetic confusions, and cascaded ASR-MT errors compound. The benchmark is publicly released.
- First Telugu spoken factoid QA benchmark: 2,001 pairs, six domains.
- Gemini-as-a-judge best matches human ratings but is unevenly strict.
- Open-weight judges penalize correct answers differing in surface form.
- Speech input introduces phonetic confusions; ASR-MT cascades compound errors.
- AI models
- Gemini
From the original article
1 min read at huggingface.co
Question answering has advanced rapidly with large language models, but predominantly for high-resource languages, in both text and spoken settings. Spoken question answering (SQA) benchmark for Telugu remains unexplored, and the reliability of automatic evaluation in this setting remains unquantified. We introduce VākQA, a Telugu SQA benchmark of 2,001 factoid question-answer pairs across six domains, with 2.
By Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.