WorldSonus: Bringing Sound to Worlds
WorldSonus adds real-time, controllable spatial audio to silent world models with streaming diffusion.
WorldSonus is an interactive video-to-audio framework for real-time spatial sound in world models. A streaming causal autoregressive diffusion model generates audio chunks at a real-time factor of 0.41. Chunk-indexed prompt scheduling supports mid-stream sound instructions, and stereo plus ambisonic supervision aligns audio with scene geometry and camera motion. The authors report it matches or beats bidirectional models on open-domain video-to-audio benchmarks.
- Streaming causal diffusion synthesizes audio at a 0.41 real-time factor.
- Chunk-indexed prompts allow sound events to change during generation.
- Stereo and ambisonic data supervise spatially aligned audio.
- It matches or outperforms bidirectional models on open video-to-audio benchmarks.
From the original article
1 min read at huggingface.co
Recent advances in world models have enabled increasingly realistic visual synthesis. However, these generated environments remain largely silent. Bringing sound to world models poses three core challenges: real-time generation to keep pace with interactive video streams, interactive control to respond to mid-stream sound instructions, and spatially aligned stereo to reflect scene geometry and camera motion.
By Pengjun Fang, Jingyi Fa, Kam Man Wu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.