Leveraging Unlabeled Data for Emotion Recognition With Enhanced Collaborative Semi-Supervised Learning
Authors:
Open Access
- 24 April 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 6, 22196-22209
Abstract
One of the major obstacles that has to be faced when applying automatic emotion recognition to realistic human–machine interaction systems is the scarcity of labelled data for training a robust model. Motivated by this concern, this article seeks to utmost exploit unlabelled data that are pervasively available in the real-world and easy to be collected, by means of novel Semi-Supervised Learning (SSL) approaches. Conventional SSL methods such as self-training, suffer from their inherent drawback of error accumulation, i. e., the samples that are misclassified by the system are continuously employed to train the model in the following learning iterations. To address this major issue, we first propose an enhanced learning strategy, by which we re-evaluate the previously automatically labelled samples in each learning iteration, in order to update the training set by correcting the mislabelled samples. We further exploit multiple modalities and models in the SSL system, by using collaborative SSL, where all modalities and models are considered simultaneously; samples are selected by means of minimising the joint entropy. This strategy is supposed to not only improve the performance of the model for data annotation and consequently enhance the trustability of the automatically labelled data, but also to elevate the diversity of selected data. To evaluate the effectiveness of the proposed approaches, we performed extensive experiments on the RECOLA database, which includes multimodal recordings of spontaneous affective interactions of dyads. The empirical results show that the proposed approaches significantly outperform recently well-established SSL methods.Keywords
Funding Information
- Economic and Social Research Council (HJ-253479)
- H2020 European Research Council (645094)
This publication has 44 references indexed in Scilit:
- Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological dataPattern Recognition Letters, 2015
- Using unlabeled data to improve classification of emotional states in human computer interactionJournal on Multimodal User Interfaces, 2013
- SMART: Semi-Supervised Music Emotion Recognition with Social Tagging2013
- A survey of multi-view machine learningNeural Computing and Applications, 2013
- Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challengeSpeech Communication, 2011
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- When Semi-supervised Learning Meets Ensemble Learning2009
- SETRED: Self-training with Editing2005
- Semisupervised learning of classifiers: theory, algorithms, and their application to human-computer interactionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2004
- Emotion recognition in human-computer interactionIEEE Signal Processing Magazine, 2001