Leveraging Chain of Thought towards Empathetic Spoken Dialogue without Corresponding Question-Answering Data
Authors:
- 6 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Empathetic dialogue is crucial for natural human-computer interaction, allowing the dialogue system to respond in a more personalized and emotionally aware manner, improving user satisfaction and engagement. The emergence of large language models (LLMs) has revolutionized dialogue generation by harnessing their powerful capabilities and shown its potential in multimodal domains. Many studies have integrated speech with text-based LLMs to take speech question as input and output text response. However, the lack of spoken question-answering datasets that include speech style information to supervised fine-tuning (SFT) limits the performance of these systems. As a result, while these systems excel at understanding speech content, they often struggle to generate empathetic responses. In response, we propose a novel approach that circumvents the need for question-answering data, called Listen, Perceive, and Express (LPE). Our method employs a two-stage training process, initially guiding the LLM to listen the content and perceive the emotional aspects of speech. Subsequently, we utilize Chain-of-Thought (CoT) prompting to unlock the model’s potential for expressing empathetic responses based on listened spoken content and perceived emotional cues. We employ experiments to prove the effectiveness of proposed method. To our knowledge, this is the first attempt to leverage CoT for speech-based dialogue.Keywords
All Related Versions
Funding Information
- Research and Development
- National Natural Science Foundation of China
This publication has 17 references indexed in Scilit:
- LLM-based NLG Evaluation: Current Status and ChallengesComputational Linguistics, 2025
- LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents2024
- E-Chat: Emotion-Sensitive Spoken Dialogue System with Large Language Models2024
- Multi-Task Learning in Natural Language Processing: An OverviewACM Computing Surveys, 2024
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment2023
- Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models2023
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation2020
- Librispeech: An ASR corpus based on public domain audio books2015
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008