Advancing AI Voice Synthesis: Integrating Emotional Expression in Multi-Speaker Voice Generation
Authors:
- 19 February 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 193-198
Abstract
Advancements in text-to-speech (TTS) synthesis have primarily focused on natural speech and speech intelligibility, but integrating nuanced emotional expressiveness and speaker variability remains a challenge, especially in dynamic environments such as customer service and in assistive speech technologies. This paper introduces a direct text input approach over conventional phoneme-first methods, such as FastSpeech, enhancing user experience. We integrate the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) along with pitch, energy, and duration in the variance adaptor of the FastSpeech 2 model to deepen the emotional expressiveness of speech. In this paper, we propose a Multi-speaker Emotional Text-to-speech Synthesis System (METTS) which allows users to input desired text, select from various speaker voices, and choose emotional tones ranging from happiness to sadness, surprise, neutrality, and anger. Unique to METTS is the feature that allows users to integrate personal voice datasets, making it highly customizable. We assess speech quality and naturalness with the NISQA model, achieving a 3.72 ± 0.78 MOS score for multi-speaker evaluation and 4.09±0.65 for individual speaker voices. The paper details METTS's architecture, enhancements to FastSpeech2, and methods for embedding emotional and speaker variations.Keywords
This publication has 17 references indexed in Scilit:
- An Overview of Affective Speech Synthesis and Conversion in the Deep Learning EraProceedings of the IEEE, 2023
- An Analysis of Acoustic Features for Attention Score in Thai MoCA Assessment2022
- ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform2022
- Varianceflow: High-Quality and Controllable Text-to-Speech using Variance Information via Normalizing Flow2022
- Towards Expressive Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis2022
- Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset2021
- StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi2017
- Opensmile2010