On granularity of prosodic representations in expressive text-to-speech
Authors:
- Mikolaj BabianskiMBMikolaj Babianski
- Kamil PokoraKPKamil Pokora
- Raahil ShahRSRaahil Shah
- Rafal SienkiewiczRSRafal Sienkiewicz
- Daniel KorzekwaDKDaniel Korzekwa
- Viacheslav KlimkovVKViacheslav Klimkov
Preprint
Abstract
In expressive speech synthesis it is widely adopted to use latent prosody representations to deal with variability of the data during training. Same text may correspond to various acoustic realizations, which is known as a one-to-many mapping problem in text-to-speech. Utterance, word, or phoneme-level representations are extracted from target signal in an auto-encoding setup, to complement phonetic input and simplify that mapping. This paper compares prosodic embeddings at different levels of granularity and examines their prediction from text. We show that utterance-level embeddings have insufficient capacity and phoneme-level tend to introduce instabilities when predicted from text. Word-level representations impose balance between capacity and predictability. As a result, we close the gap in naturalness by 90% between synthetic speech and recordings on LibriTTS dataset, without sacrificing intelligibility.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- DelightfulTTS: The Microsoft Speech Synthesis System for Blizzard Challenge 20212021
- Parallel Tacotron: Non-Autoregressive and Controllable TTS2021
- An Overview of Voice Conversion and Its Challenges: From Statistical Modeling to Deep LearningIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- CopyCat: Many-to-Many Fine-Grained Prosody Transfer for Neural Text-to-Speech2020
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech2019
- Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS2019
- Fine-Grained Robust Prosody Transfer for Single-Speaker Neural Text-To-Speech2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- In Other News: a Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data2019
- Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions2018