Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning
Kraken speech-to-speech translation model builds on Qwen3-8B with low-bitrate vector quantization and source-conditioned vocoding.
Kraken augments a pre-trained Qwen3-8B LLM with speech feature inputs and low-bitrate single-layer vector-quantized tokens trained to reconstruct SSL features. A separate token-to-waveform decoder, Autowave-X, is conditioned on source speech to improve non-linguistic transfer. Training used 150k hours of multilingual and multitask speech data. The model reportedly beats SeamlessM4T-Large v2 and Qwen2.5-Omni in translation quality and speaker/prosody transfer.
- Low-bitrate single-layer VQ tokens replace high-bitrate speech token prediction
- Autowave-X vocoder conditioned on source speech improves prosody and speaker transfer
- Trained on 150k hours of multilingual multitask speech data
- Outperforms SeamlessM4T-Large v2 and Qwen2.5-Omni on translation quality
From the original article
1 min read at arxiv.orgSpeech-to-speech translation (S2ST) has advanced significantly with speech LLMs, offering the potential for joint optimization and preserving non-linguistic information. However, these models struggle with predicting high-bitrate speech tokens in LLMs, and face the challenge of relying on S2ST training data with ideally aligned speaker identity and prosody.
By Hayato Futami · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.