Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning
Kraken speech-to-speech translation model builds on Qwen3-8B with low-bitrate vector quantization and source-conditioned vocoding.
Kraken augments a pre-trained Qwen3-8B LLM with speech feature inputs and low-bitrate single-layer vector-quantized tokens trained to reconstruct SSL features. A separate token-to-waveform decoder, Autowave-X, is conditioned on source speech to improve non-linguistic transfer. Training used 150k hours of multilingual and multitask speech data. The model reportedly beats SeamlessM4T-Large v2 and Qwen2.5-Omni in translation quality and speaker/prosody transfer.
- Low-bitrate single-layer VQ tokens replace high-bitrate speech token prediction
- Autowave-X vocoder conditioned on source speech improves prosody and speaker transfer
- Trained on 150k hours of multilingual multitask speech data
- Outperforms SeamlessM4T-Large v2 and Qwen2.5-Omni on translation quality
Full article162 words · extracted from arxiv.org · click to collapse
Speech-to-speech translation (S2ST) has advanced significantly with speech LLMs, offering the potential for joint optimization and preserving non-linguistic information. However, these models struggle with predicting high-bitrate speech tokens in LLMs, and face the challenge of relying on S2ST training data with ideally aligned speaker identity and prosody. We propose using low-bitrate tokens based on single-layer vector quantization, trained to reconstruct self-supervised learning (SSL) features. We also employ a separate token-to-waveform decoder named Autowave-X, which is also conditioned on the source speech to improve non-linguistic transfer, thereby relaxing the training data constraints. With the integration of these techniques, we propose an S2ST model named Kraken, which augments a pre-trained LLM with speech feature inputs and the low-bitrate token outputs, followed by Autowave-X vocoder. We built the model upon Qwen3-8B and trained it using 150k hours of multilingual and multitask speech data. We demonstrated that our model exhibited better translation quality than SeamlessM4T-Large v2 and Qwen2.5-Omni, along with improved speaker and prosody transfer capabilities.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.13045