UniAVC trains one model for joint audio-video creation and editing
Original title: UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
Each branch first learns from unimodal data; scarce paired data is reserved for cross-modal coupling.
UniAVC aims to cover text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing from one set of weights. The authors argue paired corpora are too scarce to spend on skills that unimodal data can teach. Modality-decoupled schedulers warp noise independently from a shared timestep, and subtasks with one clean modality supervise asymmetric workflows, including intent-aware editing and post-inference self-refinement, without task-specific adapters.
- Video needs prolonged high noise; sparser audio can settle earlier.
- A uniform noise schedule can compromise those different dynamics.
- Intent-aware editing keeps one modality intact while constraining the other.
- Self-refinement regenerates only an unsatisfactory branch against an accepted one.
From the original article
1 min read at huggingface.co
Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing.
By Kien T. Pham, Yuhao Liu, Chong Hou Choi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.