EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion
Open Access
- 23 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 1650-1661
Abstract
Recently, the field of Text-to-Speech (TTS) has been dominated by one-stage text-to-waveform models which have significantly improved speech quality compared to two-stage models. In this work, we propose EfficientTTS 2 (EFTS2), a one-stage high-quality end-to-end TTS framework that is fully differentiable and highly efficient. Our method adopts an adversarial training process, with a differentiable aligner and a hierarchical-VAE-based waveform generator. These design choices free the model from the use of external aligners, invertible structures, and complex training procedures as most previous TTS works have. Moreover, we extend EFTS2 to the voice conversion (VC) task and propose EFTS2-VC, an end-to-end VC model that allows high-quality speech-to-speech conversion. Experimental results suggest that the two proposed models achieve better or at least comparable speech quality compared to baseline models, while also providing faster inference speeds and smaller model sizes.Keywords
This publication has 16 references indexed in Scilit:
- NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level QualityIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling2021
- Wave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis2021
- Flow-TTS: A Non-Autoregressive Network for Text to Speech Based on Flow2020
- Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram2020
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- LPCNET: Improving Neural Speech Synthesis through Linear Prediction2019
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi2017
- Tacotron: Towards End-to-End Speech Synthesis2017