Speech event detector: Timestamped detection of non-verbal vocalizations and background sounds for rich speech transcription
A 5.8M model timestamps 54 non-speech sounds and merges them into Whisper transcripts.
Researchers introduce Speech Event Detector, a 5.8-million-parameter convolutional-Transformer that scores 54 non-verbal and background sound classes every 40 milliseconds and merges timestamped events into a Whisper transcript. Labels from seven public datasets were unified with rule-based mapping and CTC forced alignment, recovering timestamps for 6,360 of 8,620 sounds, then mixed with about 168 hours of real recordings plus synthetic noise. On held-out recordings, a sensitive profile covers 67% of 9,367 events at 0.30 macro precision; a balanced profile reaches macro F1 of 0.33. Inference is about 39 times faster than real time on eight CPU threads.
- 5.8M-parameter convolutional-Transformer emits class probabilities every 40 ms.
- Covers 20 speaker vocalizations and 34 background sound classes.
- Trained on about 168 hours from seven public datasets plus synthetic mixtures.
- Sensitive profile covers 67% of 9,367 held-out events at 0.30 macro precision.
- Runs about 39 times faster than real time on eight CPU threads.
- Products
- Whisper
- AI models
- Speech Event DetectorWhisper
From the original article
1 min read at huggingface.co
Automatic speech recognition produces words but drops the non-verbal content of a recording: the breaths, coughs, sighs and laughter of the speaker, and the horns, music or barking around them. We describe Speech Event Detector, a 5.
By Ayush Kumar Bar, Promila Ghosh · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.