InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
Authors:
- Dongchao YangDYDongchao Yang
- Songxiang LiuSLSongxiang Liu
- Rongjie HuangRHRongjie Huang
- Chao WengCWChao Weng
- Helen MengHMHelen Meng
Preprint
Abstract
Expressive text-to-speech (TTS) aims to synthesize different speaking style speech according to human's demands. Nowadays, there are two common ways to control speaking styles: (1) Pre-defining a group of speaking style and using categorical index to denote different speaking style. However, there are limitations in the diversity of expressiveness, as these models can only generate the pre-defined styles. (2) Using reference speech as style input, which results in a problem that the extracted style information is not intuitive or interpretable. In this study, we attempt to use natural language as style prompt to control the styles in the synthetic speech, e.g., "Sigh tone in full of sad mood with some helpless feeling". Considering that there is no existing TTS corpus which is proper to benchmark this novel task, we first construct a speech corpus, whose speech samples are annotated with not only content transcriptions but also style descriptions in natural language. Then we propose an expressive TTS model, named as InstructTTS, which is novel in the sense of following aspects: (1) We fully take the advantage of self-supervised learning and cross-modal metric learning, and propose a novel three-stage training procedure to obtain a robust sentence embedding model, which can effectively capture semantic information from the style prompts and control the speaking style in the generated speech. (2) We propose to model acoustic features in discrete latent space and train a novel discrete diffusion probabilistic model to generate vector-quantized (VQ) acoustic tokens rather than the commonly-used mel spectrogram. (3) We jointly apply mutual information (MI) estimation and minimization during acoustic model training to minimize style-speaker and style-content MI, avoiding possible content and speaker information leakage from the style prompt.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersIEEE Transactions on Audio, Speech and Language Processing, 2025
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Prompttts: Controllable Text-To-Speech With Text Descriptions2023
- Diffsound: Discrete Diffusion Model for Text-to-Sound GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Speech Synthesis With Mixed EmotionsIEEE Transactions on Affective Computing, 2022
- SoundStream: An End-to-End Neural Audio CodecIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Class-level spectral features for emotion recognitionSpeech Communication, 2010
- A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environmentsSpeech Communication, 2008
- Recognition of Emotion From Vocal CuesArchives of General Psychiatry, 1986