Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation
StereoBind links visual motion to stereo audio using a new 29K-clip dataset and spatial-consistency benchmark.
StereoBind is a joint video-audio generation framework aimed at dynamic spatial correspondence, so perceived sound location follows the motion of its visual source. It coordinates motion tracks through visual motion binding, a spatial track encoder, and residual track RoPE. The authors introduce StereoWorld-29K, a stereo audio-video dataset with paired motion tracks, and StereoWorldBench for spatial consistency. Experiments report substantially better stereo spatial alignment than existing models while preserving overall audiovisual quality.
- StereoBind binds visual source motion to stereo sound generation.
- Three modules model source correspondence, absolute position, and relative motion.
- StereoWorld-29K supplies paired stereo audio, video, and motion tracks.
- Authors report better spatial alignment without losing audiovisual quality.
From the original article
1 min read at huggingface.co
Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/VR and interactive gaming further require stereo audio to provide an immersive sense, which remains largely overlooked. Effective stereo audio requires the perceived sound location to evolve consistently with the motion of its corresponding visual source.
By Hanmo Chen, Chengcheng Liu, Tianxiao Chen · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.