HelixWorld: A Real-time Interactive Audio-Visual World Model
HelixWorld is a real-time audio-visual world model running joint rollouts at 24 FPS.
HelixWorld is a real-time interactive world model in which visual scenes and camera-grounded spatial stereo audio evolve together under user control. The authors curate a spatial audio-visual dataset with true stereo and metric camera poses, pre-train a bidirectional teacher on 6-DoF trajectories and actions, then distill it into a few-step streaming student. The student sustains drift-free joint audio-visual rollouts at 24 FPS on one GPU. HelixBench evaluates spatial-acoustic consistency; the model matches strong silent world models visually while improving camera-aligned sound.
- Scenes and camera-grounded stereo sound co-evolve under user interaction.
- A bidirectional teacher is distilled into a few-step streaming student.
- Joint audio-visual rollouts run at 24 FPS on a single GPU.
- HelixBench tests whether sound fields track dynamic viewpoint motion.
- Products
- HelixBench
- AI models
- HelixWorld
From the original article
1 min read at huggingface.co
World simulation is inherently multisensory, demanding synchronized visual and acoustic dynamics in real time. Yet prevailing interactive world models remain strictly silent, focusing exclusively on visual rendering and control while overlooking the acoustic dimension.
By Lei Ke, Jiahao Pan, Zeyue Tian · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.