CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generation
CMA-OT aligns a music generator's latent features with hierarchical expert representations via curriculum learning and scale-aware optimal transport, improving dance-to-music quality.
CMA-OT introduces curriculum-guided multi-scale representation alignment with scale-aware optimal transport for dance-to-music generation. An external music expert provides hierarchical supervision over the generator's latent features, progressively transferring musical knowledge for stable representation learning. The optimal transport mechanism handles temporal mismatch and semantic variation across expert scales. Experiments on two datasets show state-of-the-art rhythmic synchronization, perceptual quality, and overall music generation.
- External music expert supervises generator latent features hierarchically
- Curriculum strategy progressively transfers musical knowledge during training
- Scale-aware optimal transport aligns features under temporal mismatch
From the original article
1 min read at arxiv.orgDance-to-music (D2M) generation aims to synthesize music that is rhythmically and stylistically aligned with dance videos. A key challenge arises from the semantic mismatch between sparse dance cues, such as rhythm and style, and the dense information required for music composition, including structure, instrumentation, and expressive dynamics.
By Jinting Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.