EMO-Codec: An In-Depth Look at Emotion Preservation Capacity of Legacy and Neural Codec Models with Subjective and Objective Evaluations
Authors:
- 3 December 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Neural codecs reduce speech data transmission latency and serve as the underlying tokenizer for speech language models (speech LMs). Preserving emotional information in codes is crucial for effective communication and contextual understanding. However, there is a lack of studies on emotion loss in existing codecs. This paper evaluates neural and legacy codecs using subjective and objective methods on emotion datasets like IEMOCAP. Our study identifies which codecs best preserve emotional information at various bitrates. We found that training a codec with both English and Chinese data had limited success in retaining emotional information in Chinese. Additionally, resynthesizing speech through these codecs degrades the performance of speech emotion recognition (SER), especially for emotions such as sadness, depression, fear, and disgust. Human listening tests confirmed these findings. This work guides the development of future speech technology to ensure that new codecs maintain the integrity of emotional information in speech.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Open-Emotion: A Reproducible EMO-Superb For Speech Emotion Recognition Systems2024
- Dynamic-Superb: Towards a Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark For Speech2024
- FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec2024
- Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec2023
- DESCo: Detecting Emotions from Smart Commands2022
- Speech Emotion Recognition Using Self-Supervised Features2022
- Warp-Q: Quality Prediction for Generative Neural Speech Codecs2021
- Class-Balanced Loss Based on Effective Number of Samples2019
- Effects of band reduction and coding on speech emotion recognition2016
- Rethinking the Inception Architecture for Computer Vision2016