RubatoCodec adds budgeted adaptive speech tokens and streaming decode
Original title: RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding
Code and weights are public; adaptive allocation beat fixed-rate coding on reconstruction and intelligibility at low event rates.
RubatoCodec is a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Adaptive Firing splits speech into variable-duration events from codec features and waveform spectra, without a learned boundary predictor. Each event has discrete codes and an explicit duration so a generator can predict and decode one event at a time. A multilingual model is competitive on English, Japanese, and Mandarin, and a matched text-to-speech comparison reports lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity.
- The codec is built on X-Codec-2 and uses an exact utterance-level event budget.
- Adaptive allocation outperformed fixed-rate coding at low event rates.
- Best intelligibility among compared codecs on evaluated English and Japanese benchmarks.
- The speech generator learns the event budget without an explicit generation-time rate constraint.
- Code and weights are released on GitHub and Hugging Face.
- Products
- X-Codec-2
- Organizations
- third-intelligence
- AI models
- RubatoCodec
From the original article
1 min read at huggingface.co
Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information.
By Chihiro Arata, Shohei Taniguchi, Gouki Minegishi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.