Affective Video Content Analysis With Adaptive Fusion Recurrent Network
- 22 November 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 22 (9) , 2454-2466
Abstract
Affective video content analysis is an important research topic in video content analysis and has extensive applications. Intuitively, multimodal features can depict elicited emotions, and the accumulation of temporal inputs influences the viewer's emotion. Although a number of research works have been proposed for this task, the adaptive weights of modalities and the correlation of temporal inputs are still not well studied. To address these issues, a novel framework is designed to learn the weights of modalities and temporal inputs from video data. Specifically, three network layers are designed, including statistical-data layer to improve the robustness of data, temporal-adaptive-fusion layer to fuse temporal inputs, and multimodal-adaptive-fusion layer to combine multiple modalities. In particular, the feature vectors of three input modalities are respectively extracted from three pre-trained convolutional neural networks and then fed to three statistical-data layers. Then, the output vectors of these three statistical-data layers are separately connected to three recurrent layers, and the corresponding outputs are fed to a fully-connected layer which shares parameters across modalities and temporal inputs. Finally, the outputs of the fully-connected layer are fused by the temporal-adaptive-fusion layer and then combined by the multimodal-adaptive-fusion layer. To discover the correlation of both multiple modalities and temporal inputs, adaptive weights of modalities and temporal inputs are introduced into loss functions for model training, and these weights are learned by an optimization algorithm. Extensive experiments are conducted on two challenging datasets, which demonstrate that the proposed method achieves better performances than baseline and other state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (61976159, 61622115, 61962003)
- Shanghai Engineering Research Center of Industrial Vision Perception & Intelligent Computing (17DZ2251600)
This publication has 30 references indexed in Scilit:
- Deep learningNature, 2015
- Video Affective Content Analysis: A Survey of State-of-the-Art MethodsIEEE Transactions on Affective Computing, 2015
- LIRIS-ACCEDE: A Video Database for Affective Content AnalysisIEEE Transactions on Affective Computing, 2015
- Caffe2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchersCognition and Emotion, 2010
- A Duality Based Approach for Realtime TV-L 1 Optical Flow2007
- The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data2007
- Early versus late fusion in semantic video analysis2005
- Long Short-Term MemoryNeural Computation, 1997