StepAudio 3 Gen Technical Report
StepAudio 3 Gen unifies TTS, voice design, music, and sound effects via discrete autoregressive modeling over RVQ tokens.
StepAudio 3 Gen is a general-purpose audio generation model covering zero-shot TTS, voice design, vocal generation, sound effects, music, vibe speech, and mixed audio in one framework. It uses discrete autoregressive modeling over residual vector quantization (RVQ) tokens rather than the diffusion Transformer paradigm, with a StepAudio Tokenizer representing audio at 12.5 Hz in a shared 16x2048 residual code space. Key design principles include interference-aware progressive pretraining, an RVQ Adaptor for multi-codebook acoustic representations, and shared discrete autoregressive modeling. The model reports state-of-the-art performance on TTS and voice design while retaining strong generation across speech, vocals, sound effects, and music.
- Unified model handles TTS, voice design, vocals, sound effects, music, and mixtures.
- Discrete autoregressive generation over RVQ tokens replaces diffusion Transformer approach.
- StepAudio Tokenizer uses 12.5 Hz, shared 16x2048 residual code space for semantic and acoustic features.
- Reports state-of-the-art TTS and voice design results.
From the original article
1 min read at huggingface.co
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework.
By Bin Lin, Bo Zhao, Boyang Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.