Falcon cuts causal AV speaker extraction cost on new RealSSA benchmark
Original title: Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Falcon reports 91.9% fewer parameters and 11.1 times lower GPU latency than causal AV-TFGridNet while topping RealSSA-Sim extraction quality.
The paper releases RealSSA, an audio-visual target speaker extraction benchmark with simulated scenes of 1 to 5 near-field interferers and optional 10 to 15 far-field speakers, plus a held-out real-recorded set. Falcon is a causal method whose separator does multi-scale time-frequency separation in one encoder-decoder pass, trained with an encoder-space speaker contrastive loss. The authors say Falcon reaches state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix, with large cuts versus causal AV-TFGridNet. Code and a demo are linked.
- RealSSA-Sim covers 1–5 dynamic near-field interfering speakers.
- Far-field scenes add 10–15 far-field interfering speakers.
- Parameters fall 91.9% and computation 20.2 times versus causal AV-TFGridNet.
- GPU inference latency is reported 11.1 times lower than that baseline.
- Products
- RealSSAAV-TFGridNet
- AI models
- Falcon
From the original article
1 min read at huggingface.co
Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes.
By Wendi Sang, Kai Li, Yifan Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.