AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Open-source speech foundation model AuK unifies generation and editing, trained on 1.95 million hours, with distilled AuK-Flash achieving 4.5x speedup.
AuK is an open-source foundational model that unifies speech generation and editing through natural-language instructions and audio context, trained on approximately 3.03 billion instruction-audio instances and 1.95 million hours of supervision across five task families including generation, content editing, and acoustic editing. It combines a multimodal LLM for semantic conditioning, a VAE jointly trained on speech, general audio, and music, and a hybrid rectified-flow Transformer using dual-stream MMDiT blocks followed by unified single-stream DiT blocks. Post-training applies human-feedback preference optimization for editing and reward-based reinforcement learning for generation, and the distilled AuK-Flash performs 4-step inference without classifier-free guidance at a 4.5x wall-clock speedup. Source code and model weights are released.
- Trained on 3.03 billion instruction-audio instances and 1.95 million hours of supervision.
- Combines multimodal LLM conditioning, a speech-audio VAE, and a hybrid rectified-flow Transformer.
- Post-training uses human-feedback preference optimization and reward-based reinforcement learning.
- Distilled AuK-Flash performs 4-step inference with a 4.5x wall-clock speedup.
- Code and model weights released for reproducibility.
Full article204 words · extracted from huggingface.co · click to collapse
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.08936