An Automatic Multimedia Likability Prediction System Based on Facial Expression of Observer
Open Access
- 3 August 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 110421-110434
Abstract
Every individual's perception of multimedia content varies based on their interpretation. Therefore, it is quite challenging to predict likability of any multimedia just based on its content. This paper presents a novel system for analysis of facial expressions of subject against the multimedia content to be evaluated. First, we developed a dataset by recording facial expressions of subjects under uncontrolled environment. These subjects are volunteers recruited to watch the videos of different genre, and provide their feedback in terms of likability. Subject responses are divided into three categories: Like, Neutral and Dislike. A novel multimodal system is developed using the developed dataset. The model learns feature representation from data based on the three provided categories. The proposed system contains ensemble of time distributed convolutional neural network, 3D convolutional neural network, and long short term memory networks. All the modalities in proposed architecture are evaluated independently as well as in distinct combinations. The paper also provides detailed insight into learning behavior of the proposed system.Keywords
Funding Information
- Deanship of Scientific Research at King Khalid University for funding this work through General Research Project (RGP.1/376/42)
This publication has 49 references indexed in Scilit:
- Multimodal interaction: A reviewPattern Recognition Letters, 2014
- Simultaneous Facial Feature Tracking and Facial Expression RecognitionIEEE Transactions on Image Processing, 2013
- Challenges in Representation Learning: A Report on Three Machine Learning Contests2013
- Learning spatiotemporal features by using independent component analysis with application to facial expression recognitionNeurocomputing, 2012
- Measurable Emotions: How Television Ads Really WorkJournal of Advertising Research, 2010
- AFFECTIVE CHARACTERIZATION OF MOVIE SCENES BASED ON CONTENT ANALYSIS AND PHYSIOLOGICAL CHANGESInternational Journal of Semantic Computing, 2009
- Facial expression recognition based on Local Binary Patterns: A comprehensive studyImage and Vision Computing, 2009
- Automatic Recognition of Facial Actions in Spontaneous ExpressionsJournal of Multimedia, 2006
- Diversity creation methods: a survey and categorisationInformation Fusion, 2005
- Recognizing action units for facial expression analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001