A Neural Network Architecture for Children’s Audio–Visual Emotion Recognition
Open Access
- 7 November 2023
- journal article
- research article
- Published by MDPI AG in Mathematics
- Vol. 11 (22) , 4573
Abstract
Detecting and understanding emotions are critical for our daily activities. As emotion recognition (ER) systems develop, we start looking at more difficult cases than just acted adult audio–visual speech. In this work, we investigate the automatic classification of the audio–visual emotional speech of children, which presents several challenges including the lack of publicly available annotated datasets and the low performance of the state-of-the art audio–visual ER systems. In this paper, we present a new corpus of children’s audio–visual emotional speech that we collected. Then, we propose a neural network solution that improves the utilization of the temporal relationships between audio and video modalities in the cross-modal fusion for children’s audio–visual emotion recognition. We select a state-of-the-art neural network architecture as a baseline and present several modifications focused on a deeper learning of the cross-modal temporal relationships using attention. By conducting experiments with our proposed approach and the selected baseline model, we observe a relative improvement in performance by 2%. Finally, we conclude that focusing more on the cross-modal temporal relationships may be beneficial for building ER systems for child–machine communications and environments where qualified professionals work with children.Keywords
All Related Versions
Funding Information
- Russian Science Foundation (22-45-02007)
This publication has 50 references indexed in Scilit:
- EmoReact: a multimodal approach and dataset for recognizing emotional responses in children2016
- CHEAVD: a Chinese natural emotional audio–visual databaseJournal of Ambient Intelligence and Humanized Computing, 2016
- SingaKids-Mandarin: Speech Corpus of Singaporean Children Speaking Mandarin Chinese2016
- Learning Deep Features for Discriminative Localization2016
- Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity RecognitionSensors, 2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- Survey on audiovisual emotion recognition: databases, features, and data fusion strategiesAPSIPA Transactions on Signal and Information Processing, 2014
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- The NIMH Child Emotional Faces Picture Set (NIMH‐ChEFS): a new set of children's facial emotion stimuliInternational Journal of Methods in Psychiatric Research, 2011
- Inference of attitudes from nonverbal communication in two channels.Journal of Consulting Psychology, 1967