WorldSonus: Bringing Sound to Worlds
WorldSonus adds real-time, controllable spatial stereo audio to interactive world-model video streams.
WorldSonus is an interactive video-to-audio framework for adding real-time spatial sound to world models. It uses a streaming causal autoregressive diffusion model that generates audio chunks at a real-time factor of 0.41, with chunk-indexed prompt scheduling for mid-generation sound control. Stereo supervision from stereo and ambisonic data keeps audio aligned with scene geometry and camera motion. The authors report that it matches or outperforms state-of-the-art bidirectional models on open-domain video-to-audio benchmarks for acoustic quality and spatial alignment.
- Streaming causal autoregressive diffusion synthesizes audio at RTF 0.41.
- Chunk-indexed prompts allow mid-stream control of sound events.
- Stereo and ambisonic supervision align audio with scene geometry and camera motion.
- Matches or beats bidirectional video-to-audio models on open-domain benchmarks.
From the original article
1 min read at arxiv.orgRecent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion.
By Pengjun Fang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.