Zero-Shot Voice Cloning Text-to-Speech for Dysphonia Disorder Speakers
Open Access
- 2 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 63528-63547
Abstract
Deep neural network based text-to-speech (TTS) technology has brought advances in speech synthesis approaching the quality of human speech sounds. Zero-shot voice cloning TTS is a system that accepts input in the form of text and a few seconds of a sample of the target speaker’s voice to produce speech sound waves similar to the target speaker’s voice. Some of the latest zero-shot voice cloning TTS studies still focuses on normal human voices. However, this technology still has limitations for individuals with speech disorders such as dysphonia. We observe that our baseline zero-shot TTS model applied to the dysphonia domain still has poor performance on the following aspects: speaker similarity, intelligibility or clarity of speech, and speech sound quality. This research develops 24 zero-shot voice cloning TTS models to observe which models can improve the baseline model performance on the dysphonia domain. We propose four categories to change the baseline model architecture and setting: input-level text sequences (grapheme, phoneme, or combination of grapheme-phoneme), speaker embedding type (speaker encoder or speaker model), speaker embedding position (at TTS encoder of at TTS encoder and decoder), and loss function (without or with speaker consistency loss). The experimental results show that the best model is the one uses the following configuration settings: a combination of grapheme-phoneme-level text sequences, speaker model as the speaker embedding, placing the speaker embedding at the TTS encoder only, and adding speaker consistency loss to the frame-level speech loss. Compared to the baseline model, our proposed best model is able to improve speaker cosine similarity (COS), speech intelligibility (CER), and speech sound quality (MOS) performance in the domain of dysphonia speech disorders by 0.197, 0.55%, and 0.244, respectively. When compared with the original voice of dysphonia disorder speakers, the best model also increases the speech intelligibility and quality of the speech sounds by 13.45% and 0.22, respectively.Keywords
Funding Information
- Faculty of Computer Science, Universitas Indonesia (NKB-008/UN2.F11.D/HKP.05.00/2023)
This publication has 31 references indexed in Scilit:
- TIMIT-TTS: A Text-to-Speech Dataset for Multimodal Synthetic Media DetectionIEEE Access, 2023
- Phonetic posteriorgram-based voice conversion system to improve speech intelligibility of dysarthric patientsComputer Methods and Programs in Biomedicine, 2022
- Transfer Learning, Style Control, and Speaker Reconstruction Loss for Zero-Shot Multilingual Multi-Speaker Text-to-Speech on Low-Resource LanguagesIEEE Access, 2022
- Margin Matters: Towards More Discriminative Deep Neural Network Embeddings for Speaker Recognition2019
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018
- Librispeech: An ASR corpus based on public domain audio books2015
- Audiovisual speech synthesis: An overview of the state-of-the-artSpeech Communication, 2015
- Deep neural networks for small footprint text-dependent speaker verification2014
- Text-to-Speech Synthesis2009
- Is word error rate a good indicator for spoken language understanding accuracy2004