SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model
SepGen jointly generates video, a mixed soundtrack, and per-source audio stems, and can also separate mixes from captions.
SepGen extends a pretrained audio-video generator so one sampling run emits the video, the mixed soundtrack, and one waveform per captioned source. Generation mode uses source captions to specify each stem, such as one stem per speaker in a two-speaker dialogue. Separation mode keeps the input mix unchanged and uses captions to say what to extract, including from real recordings of speech, music, and sound effects. Given a mix and captions that include spoken lines, SepGen outperforms language-conditioned separators, most clearly on speech, and still leads when those lines are removed from the captions.
- SepGen emits video, a mixed soundtrack, and one waveform per captioned source.
- Generation mode assigns each stem from a source caption, including per-speaker dialogue.
- Separation mode extracts sources from a clean mix using free-text captions.
- It beats language-conditioned separators, especially on speech, even without spoken lines in captions.
From the original article
1 min read at huggingface.co
A 4D audio-visual scene comprises a video, the dynamic geometry it depicts, and the sound sources that populate it, each with its own position and trajectory. Rendering such a scene from a novel viewpoint requires that every source be available as an individual waveform, so that it can be localized in the scene and propagated to the observer before the signals are mixed.
By Aviad Dahan, Rajaei Khatib, Yonatan Bitton · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.