TaoMate-HD streams high-resolution audio-video with progressive denoising
Original title: TaoMate-HD: Progressive-Resolution Autoregressive Audio-Video Generation
Authors report a 1.6x speedup versus direct high-resolution generation and an 11.18-point drop in character error rate.
TaoMate-HD is an autoregressive model for streaming digital humans that keep speaking as new text arrives. Each short chunk uses two low-resolution denoising steps for coarse video structure, then a high-resolution refine step, while audio is generated along the same trajectory. History is passed between chunks so the model can infer what has already been spoken. Continuation training with asymmetric text windows cut character error rate by 11.18 percentage points, and the authors say the model supports hour-long generation.
Unconfirmed: Speed and error-rate figures are the authors' reported experimental results.
- Two low-resolution denoising steps set coarse video structure per chunk.
- A final step refines video to high resolution while audio continues.
- Direct high-resolution generation is 1.6 times slower on the same hardware.
- Stage-parallel deployment adds a further 1.9x speedup over serial generation.
- Continuation training cuts character error rate by 11.18 percentage points.
From the original article
1 min read at huggingface.co
Streaming digital humans produce detailed video while continuing speech as new text arrives. High-resolution denoising is costly, and text updates can cause repeated or interrupted speech. We present TaoMate-HD, an autoregressive model for high-resolution audio-video streaming.
By Qijun Gan, Chenwei Zhang, Xinyang Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.