METTS: Multilingual Emotional Text-to-Speech by Cross-speaker and Cross-lingual Emotion Transfer
Authors:
- Xinfa ZhuXZXinfa Zhu
- Yi LeiYLYi Lei
- Tao LiTLTao Li
- Yongmao ZhangYZYongmao Zhang
- Hongbin ZhouHZHongbin Zhou
- Heng LuHLHeng Lu
- Lei XieLXLei Xie
Preprint
Abstract
Previous multilingual text-to-speech (TTS) approaches have considered leveraging monolingual speaker data to enable cross-lingual speech synthesis. However, such data-efficient approaches have ignored synthesizing emotional aspects of speech due to the challenges of cross-speaker cross-lingual emotion transfer - the heavy entanglement of speaker timbre, emotion, and language factors in the speech signal will make a system produce cross-lingual synthetic speech with an undesired foreign accent and weak emotion expressiveness. This paper proposes the Multilingual Emotional TTS (METTS) model to mitigate these problems, realizing both cross-speaker and cross-lingual emotion transfer. Specifically, METTS takes DelightfulTTS as the backbone model and proposes the following designs. First, to alleviate the foreign accent problem, METTS introduces multi-scale emotion modeling to disentangle speech prosody into coarse-grained and fine-grained scales, producing language-agnostic and language-specific emotion representations, respectively. Second, as a pre-processing step, formant shift-based information perturbation is applied to the reference signal for better disentanglement of speaker timbre in the speech. Third, a vector quantization-based emotion matcher is designed for reference selection, leading to decent naturalness and emotion diversity in cross-lingual synthetic speech. Experiments demonstrate the good design of METTS.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Do Prosody Transfer Models Transfer Prosody?2023
- CROSSSPEECH: Speaker-Independent Acoustic Representation for Cross-Lingual Speech Synthesis2023
- iEmoTTS: Toward Robust Cross-Speaker Emotion Transfer and Control for Speech Synthesis Based on Disentanglement Between Prosody and TimbreIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- GM-TCNet: Gated Multi-scale Temporal Convolutional Network using Emotion Causality for Speech Emotion RecognitionSpeech Communication, 2022
- Cross-Speaker Emotion Transfer Through Information Perturbation in Emotional Speech SynthesisIEEE Signal Processing Letters, 2022
- Cross-Speaker Emotion Disentangling and Transfer for End-to-End Speech SynthesisIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- MsEmoTTS: Multi-Scale Emotion Transfer, Prediction, and Control for Emotional Speech SynthesisIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- Formant-Based Robust Voice Activity DetectionIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- Speaker-independent emotion recognition exploiting a psychologically-inspired binary cascade classification schemaInternational Journal of Speech Technology, 2012
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004