WaveGen generates speech waveforms end to end without external codecs
Original title: WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
Scaled from 0.1B to 0.6B parameters, it reports a 1.47 word error rate and 0.75 similarity on Seed-en.
WaveGen is a diffusion text-to-waveform model that jointly learns an internal spectral trajectory and the waveform, without neural codecs, Mel vocoders, VAEs, duration predictors or self-supervised text and speech embeddings. It uses Spectral Forcing, disentangled diffusion heads and a multi-scale spectral v-loss, trained in one stage on open-source data. The authors report state-of-the-art results on the Emilia training dataset and release code and checkpoints.
Unconfirmed: State-of-the-art and benchmark figures are the authors' reported results.
- Model size was scaled from 0.1B to 0.6B parameters.
- Seed-en scores are a WER of 1.47 and a SIM of 0.75.
- Training avoids BERT embeddings, Wav2Vec 2.0 representations and REPA.
- Code and checkpoints are at github.com/sh-lee-prml/WaveGen.
From the original article
1 min read at huggingface.co
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation.
By Sang-Hoon Lee, Ha-Yeong Choi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.