EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation
EmoRES steers frozen text-to-speech models by splitting emotion vectors into shared and residual components, improving control without retraining.
EmoRES is a training-free vector-steering method that decomposes an emotion direction into a shared component, which moves speech away from neutral, and a residual component aimed at the requested emotion. On IEMOCAP it beat CoCoEmo on IndexTTS-2 and CosyVoice2, raising rank correlation by 26.13 and 12.97 points (relative gains of 118.8% and 33.1%) and emotion hit rate by 12.95 and 6.92 points. Listeners identified the dominant emotion up to 35% more often, fidelity improved up to 17.3%, and EmoRES was preferred for naturalness in up to 63.8% of pairwise comparisons.
- EmoRES is a training-free steering method for emotion-conditioned speech synthesis.
- Emotion vectors split into a shared non-neutral component and a residual emotion direction.
- On IEMOCAP, rank correlation rose 26.13 and 12.97 points versus CoCoEmo.
- Listeners identified the requested emotion up to 35% more often.
- Products
- IndexTTS-2CosyVoice2
- AI models
- IndexTTS-2CosyVoice2
From the original article
1 min read at huggingface.co
Emotion-conditioned text-to-speech (TTS) models may fail to express the requested emotion reliably, and improving controllability by additional training is costly in both computation and emotion-labeled speech training data. We therefore study vector steering, a training-free approach that modifies the internal representations of a frozen model.
By Kuan-Po Huang, Haohe Liu, Puyuan Peng · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.