CMCU-CSS: Enhancing Naturalness via Commonsense-based Multi-modal Context Understanding in Conversational Speech Synthesis
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 6081-6089
Abstract
Conversational Speech Synthesis (CSS) aims to produce speech appropriate for oral communication. However, the complexity of context dependency modeling poses significant challenges in the field of CSS, especially the mutual psychological influence between interlocutors. Previous studies have verified that prior commonsense knowledge helps machines understand subtle psychological information (e.g., feelings and intentions) in spontaneous oral dialogues. Therefore, to enhance context understanding and improve the naturalness of synthesized speech, we propose a novel conversational speech synthesis system (CMCU-CSS) that incorporates the Commonsense-based Multi-modal Context Understanding (CMCU) module to model the dynamic emotional interaction among interlocutors. Specifically, we first utilize three implicit states (intent state, internal state and external state) in CMCU to model the context dependency between inter/intra speakers with the help of commonsense knowledge. Furthermore, we infer emotion vectors from the fusion of these implicit states and multi-modal features to enhance the emotion discriminability of synthesized speech. This is the first attempt to combine commonsense knowledge with conversational speech synthesis, and its effect in terms of emotion discriminability of synthetic speech is evaluated by emotion recognition in conversation task. The results of subjective and objective evaluations demonstrate that the CMCU-CSS model achieves more natural speech with context-appropriate emotion and is equipped with the best emotion discriminability, surpassing that of other conversational speech synthesis models.Keywords
Funding Information
- Fundamental Research Funds for the Central Universities (No. 2023RC13)
- The State Key Laboratory of Multimodal Artificial Intelligence Systems (No. 202200042)
- National Natural Science Foundation of China (NSFC) (No. 62271083)
This publication has 11 references indexed in Scilit:
- End-to-End Text-to-Speech Based on Latent Representation of Speaking Styles Using Spontaneous Dialogue2022
- Acoustic Modeling for End-to-End Empathetic Dialogue Speech Synthesis Using Linguistic and Prosodic Contexts of Dialogue History2022
- Towards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis2022
- Conversational End-to-End TTS for Voice Agents2021
- ATOMIC: An Atlas of Machine Commonsense for If-Then ReasoningProceedings of the AAAI Conference on Artificial Intelligence, 2019
- DialogueRNN: An Attentive RNN for Emotion Detection in ConversationsProceedings of the AAAI Conference on Artificial Intelligence, 2019
- ICON: Interactive Conversational Memory Network for Multimodal Emotion Detection2018
- Commonsense reasoning and commonsense knowledge in artificial intelligenceCommunications of the ACM, 2015
- Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions2009
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008