A Self-Fusion Network Based on Contrastive Learning for Group Emotion Recognition
- 12 September 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. 10 (2) , 458-469
Abstract
Group emotion recognition (GER) from image has attracted much attention in recent years. Networks using attention mechanism for GER have shown great potential. However, the performance of the current attention-based GER networks suffers from the indistinctive features of individuals in the group, poor feature fusion weights, and the lack of semantic information of the objects in the image. We present a new framework that is composed of three networks, FacesNet, SceneNet, and ObjectsNet, to address these shortcomings. This new framework is designed to recognize group emotion by exploiting the information from the faces, scene, and objects in image. In FacesNet, we use contrastive learning to help the network extract distinctive emotion features and a new attention mechanism named self-fusion module to generate precise fusion weights for aggregation of individual facial features. We design SceneNet to capture the multiscale scene features to exploit the emotion cues from the scene. We construct a fully connected network named ObjectsNet to classify the semantic features of the objects. Finally, we linearly integrate the outputs of these three networks as the final output of this unique framework for GER. Experiment results on three datasets for GER show that our proposed framework achieved better performance in terms of recognition accuracy compared with the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (62173353)
- Guangzhou Municipal People–s Livelihood Science and Technology Plan (201903010040)
- Science and Technology Program of Guangzhou, China (202007030011)
This publication has 54 references indexed in Scilit:
- A Deep Neural Network-Driven Feature Learning Method for Multi-view Facial Expression RecognitionIEEE Transactions on Multimedia, 2016
- Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition2015
- FaceNet: A unified embedding for face recognition and clustering2015
- The more the merrier: Analysing the affect of a group of people in images2015
- Group AffectAnnual Review of Organizational Psychology and Organizational Behavior, 2015
- Deep Face Recognition2015
- A new approach of audio emotion recognitionExpert Systems with Applications, 2014
- Video Google: a text retrieval approach to object matching in videos2003
- Emotion recognition in human-computer interactionIEEE Signal Processing Magazine, 2001
- Affective reactions to acoustic stimuliPsychophysiology, 2000