A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space
Authors:
- 25 December 2015
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 8 (1) , 3-14
Abstract
Dimensional models have been proposed in psychology studies to represent complex human emotional expressions. Activation and valence are two common dimensions in such models. They can be used to describe certain emotions. For example, anger is one type of emotion with a low valence and high activation value; neutral has both a medium level valence and activation value. In this work, we propose to apply multi-task learning to leverage activation and valence information for acoustic emotion recognition based on the deep belief network (DBN) framework. We treat the categorical emotion recognition task as the major task. For the secondary task, we leverage activation and valence labels in two different ways, category level based classification and continuous level based regression. The combination of the loss functions from the major and secondary tasks is used as the objective function in the multi-task learning framework. After iterative optimization, the values from the last hidden layer in the DBN are used as new features and fed into a support vector machine classifier for emotion recognition. Our experimental results on the Interactive Emotional Dyadic Motion Capture and Sustained Emotionally Colored Machine-Human Interaction Using Nonverbal Expression databases show significant improvements on unweighted accuracy, illustrating the benefit of utilizing additional information in a multi-task learning setup for emotion recognition.Keywords
This publication has 30 references indexed in Scilit:
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Combining modality specific deep neural networks for emotion recognition in video2013
- AVEC 20132013
- Improving deep neural networks for LVCSR using rectified linear units and dropout2013
- Exploring Cross-Modality Affective Reactions for Audiovisual Emotion RecognitionIEEE Transactions on Affective Computing, 2013
- Combining video, audio and lexical indicators of affect in spontaneous conversation via particle filtering2012
- AVEC 20122012
- The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited AgentIEEE Transactions on Affective Computing, 2011
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Describing the emotional states that are expressed in speechSpeech Communication, 2003