Controllable Multi-Speaker Emotional Speech Synthesis With an Emotion Representation of High Generalization Capability
- 11 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 16 (1) , 68-82
Abstract
The aim of multi-speaker emotional speech synthesis is to generate speech for a designated speaker in a desired emotional state. The task is challenging due to the presence of speech variations, such as noise, content, and timbre, which can obstruct emotion extraction and transfer. This paper proposes a new approach to performing multi-speaker emotional speech synthesis. The proposed method, which is based on a seq2seq synthesizer, integrates emotion embedding as a conditioned variable to convey exact emotional information from reference audio to the synthesized speech. To dig emotion representation capability, we utilize a three-dimensional acoustic feature as input. And an emotion generalization module with adaptive instance normalization (AdaIN) is proposed to obtain emotion embedding with high generalization ability, which also results in improved controllability. The derived emotion embedding from the generalization module can be readily conditioned by affine parameters, allowing for control both the emotion category and the emotion intensity of synthesized speech. Various emotional speech synthesis experimental results of the propposed method demonstrate its state-of-the-art performance in multi-speaker emotional speech synthesis, coupled with its advantage of high emotion controllability.Keywords
Funding Information
- National Natural Science Foundation of China Joint Fund Key Project (U2003207)
- National Natural Science Foundation of China (61902064)
- Natural Science Foundation of Anhui Province (1908085MF209)
- Jiangsu Frontier Technology Basic Research Project (BK20192004)
This publication has 36 references indexed in Scilit:
- Emotional voice conversion: Theory, databases and ESDSpeech Communication, 2022
- Voice Conversion Using Spectral Mapping and TD-PSOLA2021
- Controlling Emotion Strength with Relative Attribute for End-to-End Speech Synthesis2019
- End-to-End Emotional Speech Synthesis Using Style Tokens and Semi-Supervised Training2019
- Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesisSpeech Communication, 2018
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- Emotional statistical parametric speech synthesis using LSTM-RNNs2017
- An investigation to transplant emotional expressions in DNN-based TTS synthesis2017
- Automatic Analysis of Prosody for Multi‐lingual Speech Corpora2001
- Synthesis Strategies1997