Controllable Accented Text-to-Speech Synthesis With Fine and Coarse-Grained Intensity Rendering
- 1 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 2188-2201
Abstract
Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1), which is challenging as L2 is different from L1 in terms of phonetic rendering and prosody pattern (pitch, energy, and duration variance, etc.). Accented TTS has several significant real-world applications, such as language learning, preserving and documenting endangered languages and dialects, etc. that make it an important area of research and development. Moreover, changing the accent intensity of any conversational AI system has the potential to allow specific users to understand its produced speech better. However, there is no intuitive solution for the control of the accent intensity for an utterance at both fine and coarse-grained levels, that are phoneme and utterance levels respectively. In this work, we propose a neural TTS architecture that allows us to control the accent style and its intensity. This is achieved through two novel mechanisms: 1) the front-end and back-end accent knowledge injection mechanism to enhance the accent interpretability of TTS modeling; and 2) an automatic speech recognition (ASR) based accent intensity modeling strategy to quantify the accent intensity in both L2 phoneme and utterance levels. In the front-end, a new accent variation adaptor seeks to project the accent-aware pitch, energy and duration features at a phoneme level, with the help of the fine-grained accent intensity information; In the back-end, a consistency constraint module that ensures the synthesized L2 speech manifests the expected accent intensity, is injected in the front-end, precisely. Experiments show that the proposed system attains superior performance to the baseline models in terms of accent rendering and intensity control. To our knowledge, this is the first study of accented TTS with explicit intensity control at both fine and coarse-grained levels.Keywords
Funding Information
- National Natural Science Foundation of China (62206136)
- Guangdong Provincial Key Laboratory of Human Digital Twin (2022B1212010004)
- Inner Mongolia Science and Technology Achievement Transfer and Transformation Demonstration Zone
- University Collaborative Innovation Base
- University Entrepreneurship Training Base” Construction (21300-231510)
- Internal Project Fund from Shenzhen Research Institute of Big Data (T00120220002)
- ShenZhen Science and Technology Research
- Fundamental Research Key (JCYJ20220818103001002)
This publication has 89 references indexed in Scilit:
- Computer-assisted pronunciation training—Speech synthesis is almost all you needSpeech Communication, 2022
- Accentron: Foreign accent conversion to arbitrary non-native speakers using zero-shot learningComputer Speech & Language, 2022
- Golden speaker builder – An interactive tool for pronunciation trainingSpeech Communication, 2019
- Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiersSpeech Communication, 2015
- Perceived accentedness and intelligibility: The relative contributions of F0 and durationSpeech Communication, 2013
- Automatic conversion between pronunciations of different English accentsSpeech Communication, 2011
- Foreign accent conversion in computer assisted pronunciation trainingSpeech Communication, 2009
- Dynamic Time Warping2007
- Phone-level pronunciation scoring and assessment for interactive language learningSpeech Communication, 2000
- Accents of English 31982