VocalGrad finds audio language models near chance on acoustic change
Original title: VocalGrad: Evaluating Acoustic Perception in Audio Language Models
Human annotators score high on binary temporal-change classification while models stay near chance.
VocalGrad tests whether large audio language models perceive low-level acoustic variation, framed as binary classification of temporal change direction. The authors say this isolates perception from higher-level reasoning. Results show a large gap versus human annotators, and linear probing finds two failure modes in internal representations.
Unconfirmed: Near-chance scores are author-reported benchmark results, not independently confirmed here.
- The task is binary classification of temporal change direction.
- Some categories keep task information that the language-model head does not use.
- In other categories, linear decodability falls across model layers.
From the original article
1 min read at arxiv.orgRecent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition. However, these models still struggle to reliably utilize acoustic information, particularly when it co-occurs with spoken linguistic content.
By Koki Ryu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.