LiveFoley streams video-to-audio with look-ahead below perception
Original title: LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
A trailing window and wider local diffusion keep alignment with offline models while cutting frontend latency.
LiveFoley is a streaming causal autoregressive diffusion system for real-time video-to-audio. A Trailing Sliding Window turns non-causal segment encoders into streaming modules without retraining, and a configurable hop size cuts frontend look-ahead below the human perceptual threshold. History LocDiT adds past context at no extra latency; Future-Warmup LocDiT can trade a small backend look-ahead for higher quality. The authors report matching offline temporal alignment and leading other streaming systems in generation quality.
- The Unified Varhop Framework tunes hop size for different latency and compute budgets.
- History LocDiT is the default audio backend and adds past context at no latency cost.
- Future-Warmup LocDiT uses an approximate future condition and accepts backend look-ahead.
- A project page is at dalision.github.io/livefoley-page.
From the original article
1 min read at huggingface.co
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality.
By Yusheng Dai, Vladimir Iashin, Kangdi Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.