MIST-Tacotron: End-to-End Emotional Speech Synthesis Using Mel-Spectrogram Image Style Transfer
Open Access
- 2 March 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 25455-25463
Abstract
With the development of voice synthesis technology using deep learning, voice synthesis research that expresses the characteristics and emotions of speakers is actively being conducted. Current technology does not satisfactorily express various emotions and characteristics for speakers with very low or high vocal ranges and for speakers with dialects. In this paper, we propose mel-spectrogram image transfer (MIST)-Tacotron, a Tacotron 2-based speech synthesis model that adds a reference encoder with an image style transfer module. The proposed method is a technique that adds image style transfer to the existing Tacotron 2 model and extracts the speaker’s feature from the reference mel-spectrogram using a pre-trained deep learning model. Through the extracted feature, the style such as pitch, tone, and duration of the speaker are trained to express the style and emotion of the speaker more clearly. To extract the speaker’s style independently from the speaker’s timbre and emotion, the ID value for the speaker and the ID value for the emotional state were used as inputs. Performance is evaluated by F0 voiced error (FVE), F0 gross pitch error (F0 GPE), mel-cepstral distortion (MCD), band aperiodicity distortion (BAPD), voiced/unvoiced error (VUVE), false positive rate (FPR), and false negative rate (FNR). The performance of the proposed model was observed to have lower error values than the existing models, GST (Global Style Token) Tacotron and VAE (Variational Autoencoder) Tacotron. As a result of measuring mean opinion score (MOS), the sound quality of the proposed model received the highest score in terms of emotional expression and speaker style reflection.Keywords
Funding Information
- National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT, Korea Government (2021R1F1A1064080)
This publication has 27 references indexed in Scilit:
- A Neural Parametric Singing Synthesizer Modeling Timbre and Expression from Natural SongsApplied Sciences, 2017
- Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization2017
- Tacotron: Towards End-to-End Speech Synthesis2017
- Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis2017
- Image Style Transfer Using Convolutional Neural Networks2016
- Postfilters to Modify the Modulation Spectrum for Statistical Parametric Speech SynthesisIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2016
- WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time ApplicationsIEICE Transactions on Information and Systems, 2016
- A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environmentsSpeech Communication, 2008
- A Speech Parameter Generation Algorithm Considering Global Variance for HMM-Based Speech SynthesisIEICE Transactions on Information and Systems, 2007
- Speech probability distributionIEEE Signal Processing Letters, 2003