NVAlign: Direct-Gradient Optimization for Non-Verbal Control in Continuous Autoregressive Flow Matching Text-to-Speech
NVAlign uses reward gradients to improve non-verbal tag control in flow-matching text-to-speech.
NVAlign is a direct-gradient post-training method for controlling inline non-verbal vocalization tags in continuous autoregressive flow-matching text-to-speech. The TTS model and an NVV-aware ASR model are first fine-tuned on annotated speech; the ASR model is then frozen as a reward model. A two-step gradient surrogate backpropagates that reward through the flow-matching sampler to update the autoregressive backbone and acoustic flow head, with fidelity penalties and reference-velocity regularization to preserve speaker similarity and quality. On NVV-SuperBench and human listening tests, NVAlign improves tag-following accuracy over supervised fine-tuning and Flow-GRPO.
- NVAlign post-trains continuous autoregressive flow-matching TTS for NVV tags.
- A frozen NVV-aware ASR model supplies the reward.
- A gradient surrogate updates the backbone and acoustic flow head.
- NVAlign beats SFT and Flow-GRPO on tag-following accuracy.
From the original article
1 min read at huggingface.co
While modern text-to-speech (TTS) systems generate highly natural speech and support inline non-verbal vocalization (NVV) tags, accurate control over these events remains challenging. A key gap is the lack of established post-training methods for non-verbal control in continuous autoregressive flow-matching TTS.
By Qiaolin Wang, Pedro Sandoval-Segura, Anunaya Joshi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.