Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models
Preprint
Abstract
Large language models (LLMs) have started to play a vital role in modelling speech and text. To explore the best use of context and multiple systems' outputs for post-ASR speech emotion prediction, we study LLM prompting on a recent task named GenSEC. Our techniques include ASR transcript ranking, variable conversation context, and system output fusion. We show that the conversation context has diminishing returns and the metric used to select the transcript for prediction is crucial. Finally, our best submission surpasses the provided baseline by 20% in absolute accuracy.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Speech Emotion Recognition With ASR Transcripts: a Comprehensive Study on Word Error Rate and Fusion Techniques2024
- Crossmodal ASR Error Correction With Discrete Speech Units2024
- Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting2024
- LanSER: Language-Model Supported Speech Emotion Recognition2023
- Fusing ASR Outputs in Joint Training for Speech Emotion Recognition2022
- Multi-Modal Attention for Speech Emotion Recognition2020
- Multi-Modal Learning for Speech Emotion Recognition: An Analysis and Comparison of ASR Outputs with Ground Truth Transcription2019
- Fusion Techniques for Utterance-Level Emotion Recognition Combining Speech and Transcripts2019
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition2004