Replacement-Aware Distillation for Video Dubbing
Original title: VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing
VidDub distills a teacher into a two-step student using replacement-aware cross-audio supervision.
Video dubbing synthesizes lip movements matching target speech while preserving speaker identity. Existing editors trained via self-reconstruction conflict with replaced speech. VidDub presents a full-context diffusion editor distilled into two denoising steps with replacement-aware supervision.
From the original article
1 min read at huggingface.co
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced.
By Yexin Liu, Yunlong Lin, Zehan Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.