Spontts: Modeling and Transferring Spontaneous Style for TTS
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12171-12175
Abstract
Spontaneous speaking style exhibits notable differences from other speaking styles due to various spontaneous phenomena (e.g., filled pauses, prolongation) and substantial prosody variation (e.g., diverse pitch and duration variation, occasional non-verbal speech like a smile), posing challenges to modeling and prediction of spontaneous style. Moreover, the limitation of high-quality spontaneous data constrains spontaneous speech generation for speakers without spontaneous data. To address these problems, we propose SponTTS, a two-stage approach based on neural bottleneck (BN) features to model and transfer spontaneous style for TTS. In the first stage, we adopt a Conditional Variational Autoencoder (CVAE) to capture spontaneous prosody from a BN feature and involve the spontaneous phenomena by the constraint of spontaneous phenomena embedding prediction loss. Besides, we introduce a flow-based predictor to predict a latent spontaneous style representation from the text, which enriches the prosody and context-specific spontaneous phenomena during inference. In the second stage, we adopt a VITS-like module to transfer the spontaneous style learned in the first stage to the target speakers. Experiments demonstrate that SponTTS is effective in modeling spontaneous style and transferring the style to the target speakers, generating spontaneous speech with high naturalness, expressiveness, and speaker similarity. The zero-shot spontaneous style TTS test further verifies the generalization and robustness of SponTTS in generating spontaneous speech for unseen speakers.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling2023
- WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition2022
- AISpeech-SJTU Accent Identification System for the Accented English Speech Recognition Challenge2021
- Conversational End-to-End TTS for Voice Agents2021
- Breathing and Speech Planning in Spontaneous Speech Synthesis2020
- Disfluency Insertion for Spontaneous TTS: Formalization and Proof of Concept2018
- Generalized End-to-End Loss for Speaker Verification2018
- Harvest: A High-Performance Fundamental Frequency Estimator from Speech Signals2017
- Synthesising Uncertainty: The Interplay of Vocal Effort and Hesitation Disfluencies2017
- Variability and Consistency in Speech Breathing During ReadingJournal of Speech, Language, and Hearing Research, 1994