Finding Emotions Where They Belong: Rethinking Audio Emotion Recognition through Masked Temporal Affective Grounding
Researchers recast audio emotion recognition as temporal grounding and train EMO-TAG to localize affective spans.
The paper reformulates audio emotion recognition as Temporal Affective Grounding, linking emotions to bounded speech spans rather than one label per recording. Masked Temporal Affective Grounding combines full-sequence language modeling with emotion and timestamp losses under attention masking, including a distance-aware temporal penalty. EMO-TAG, fine-tuned with this objective, is compared with Flamingo-Next, Audio-Reasoner, and AffectGPT. Those baselines remain weak at affective temporal grounding even when emotion recognition is competitive.
- Recasts audio emotion recognition as temporally bounded affective spans
- Masked training reduces shortcuts from vocal tone descriptions
- Distance-aware timestamp loss penalizes larger temporal errors
- EMO-TAG trails baselines on grounding despite competitive recognition
From the original article
1 min read at arxiv.orgAudio emotion recognition (AER) typically assigns a single label to an entire recording, leaving the temporal scope of that label ambiguous when multiple speakers and affective events are present. We address this limitation by reformulating AER as a Temporal Affective Grounding (TAG) task that associates emotions with temporally bounded speech spans and vocal tone descriptions.
By Abdelrahman Mohamed · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.