Multi-Modal Emotion Recognition by Fusing Correlation Features of Speech-Visual
Authors:
- 29 January 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 28 (10709908) , 533-537
Abstract
To effectively fuse speech and visual features, this letter proposes a multi-modal emotion recognition method by fusing correlation features of speech-visual. Firstly, speech and visual features are extracted by two-dimensional convolutional neural network (2D-CNN) and three-dimensional convolutional neural network (3D-CNN), respectively. Secondly, the speech and visual features is processed by feature correlation analysis algorithm in multi-modal fusion. In addition, the class information of speech and visual features are also applied to the feature correlation analysis algorithm, which can effectively fuse speech and visual features and improve the performance of multi-modal emotion recognition. Finally, support vector machines (SVM) completes the classification of multi-modal speech and visual emotion recognition. Experimental results on the RML, eNTERFACE05, BAUM-1 s datasets show that the recognition rate of our method is higher than other state-of-the-art methods.Keywords
Funding Information
- Chongqing Key Project of Technology Innovation and Application Development (cstc2019jscx-mbdxX0050)
- Graduate Research and Innovation Foundation of Chongqing (CYS20071)
This publication has 27 references indexed in Scilit:
- Audio-visual emotion fusion (AVEF): A deep efficient weighted approachInformation Fusion, 2019
- Joint optimisation convex‐negative matrix factorisation for multi‐modal image collection summarisation based on images and tagsIET Computer Vision, 2018
- Discriminative Multiple Canonical Correlation Analysis for Information FusionIEEE Transactions on Image Processing, 2017
- A review of affective computing: From unimodal analysis to multimodal fusionInformation Fusion, 2017
- BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental StatesIEEE Transactions on Affective Computing, 2017
- Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion RecognitionIEEE Transactions on Circuits and Systems for Video Technology, 2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Recognizing Human Emotional State From Audiovisual Signals*IEEE Transactions on Multimedia, 2008
- Lip synchronization through alignment of speech and image data1995