Multi-Rater Consensus Learning for Modeling Multiple Sparse Ratings of Affective Behaviour
Open Access
- 20 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 15 (3) , 859-871
Abstract
The use of multiple raters to label datasets is an established practice in affective computing. The principal goal is to reduce unwanted subjective bias in the labelling process. Unfortunately, this leads to the key problem of identifying a ground truth for training the affect recognition system. This problem becomes more relevant in a sparsely-crossed annotation where each rater only labels a portion of the full dataset to ensure a manageable workload per rater. In this paper, we introduce a Multi-Rater Consensus Learning (MRCL) method which learns a representative affect recognition model that accounts for each rater's agreement with the other raters. MRCL combines a multitask learning (MTL) regularizer and a consensus loss. Unlike standard MTL, this approach allows the model to learn to predict each rater's label while explicitly accounting for the consensus among raters. We evaluated our approach on two different datasets based on spontaneous affective body movement expressions for pain behaviour detection and laughter type recognition respectively. The two naturalistic datasets were chosen for the different forms of labelling (different in affect, observation stimuli, and raters) that they together offer for evaluating our approach. Empirical results demonstrate that MRCL is effective for modelling affect from datasets with sparsely-crossed multi-rater annotation.Keywords
Funding Information
- EU Future and Emerging Technologies Proactive Programme H2020 (824160)
This publication has 30 references indexed in Scilit:
- Improving Uncertainty Estimation in Convolutional Neural Networks Using Inter-rater Agreement2019
- Let’s Agree to Disagree: Learning Highly Debatable Multirater Labelling2019
- Multi-task Survival Analysis2017
- Learning from crowdsourced labeled data: a surveyArtificial Intelligence Review, 2016
- Perception and Automatic Recognition of Laughter from Whole-Body Motion: Continuous and Categorical PerspectivesIEEE Transactions on Affective Computing, 2015
- More Consensus Than IdiosyncrasyAcademic Medicine, 2014
- Multi-task and multi-view learning of user stateNeurocomputing, 2014
- Human Perception of Laughter from Context-Free Whole Body Motion Dynamic Stimuli2013
- Affective Body Expression Perception and Recognition: A SurveyIEEE Transactions on Affective Computing, 2012
- Seeing the same thing differentlyAdvances in Health Sciences Education, 2012