Controllable Emotion Transfer For End-to-End Speech Synthesis
Preprint
Abstract
Emotion embedding space learned from references is a straightforward approach for emotion transfer in encoder-decoder structured emotional text to speech (TTS) systems. However, the transferred emotion in the synthetic speech is not accurate and expressive enough with emotion category confusions. Moreover, it is hard to select an appropriate reference to deliver desired emotion strength. To solve these problems, we propose a novel approach based on Tacotron. First, we plug two emotion classifiers -- one after the reference encoder, one after the decoder output -- to enhance the emotion-discriminative ability of the emotion embedding and the predicted mel-spectrum. Second, we adopt style loss to measure the difference between the generated and reference mel-spectrum. The emotion strength in the synthetic speech can be controlled by adjusting the value of the emotion embedding as the emotion embedding can be viewed as the feature map of the mel-spectrum. Experiments on emotion transfer and strength control have shown that the synthetic speech of the proposed method is more accurate and expressive with less emotion category confusions and the control of emotion strength is more salient to listeners.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- A Comparison of Expressive Speech Synthesis Approaches based on Neural Network2018
- Fully Character-Level Neural Machine Translation without Explicit SegmentationTransactions of the Association for Computational Linguistics, 2017
- Tacotron: Towards End-to-End Speech Synthesis2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- A Neural Algorithm of Artistic StyleJournal of Vision, 2016
- Emotion transplantation through adaptation in HMM-based speech synthesisComputer Speech & Language, 2015
- Emotional transplant in statistical speech synthesis based on emotion additive model2015
- Deep Learning for Acoustic Modeling in Parametric Speech Generation: A systematic review of existing techniques and future trendsIEEE Signal Processing Magazine, 2015
- HMM-Based Emotional Speech Synthesis Using Average Emotion Model2006
- Acoustic Modeling of Speaking Styles and Emotional Expressions in HMM-Based Speech SynthesisIEICE Transactions on Information and Systems, 2005