Benchmarking Multimodal Sentiment Analysis
Authors:
Preprint
Abstract
We propose a framework for multimodal sentiment analysis and emotion recognition using convolutional neural network-based feature extraction from text and visual modalities. We obtain a performance improvement of 10% over the state of the art by combining visual, text and audio features. We also discuss some major issues frequently ignored in multimodal sentiment analysis research: the role of speaker-independent models, importance of the modalities and generalizability. The paper thus serve as a new benchmark for further research in multimodal sentiment analysis and also demonstrates the different facets of analysis to be considered while performing such tasks.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- Affective Computing and Sentiment AnalysisIEEE Intelligent Systems, 2016
- Semantic Audiovisual Data Fusion for Automatic Emotion Recognition2015
- YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual ContextIEEE Intelligent Systems, 2013
- Recognizing Affect from Linguistic Information in 3D Continuous SpaceIEEE Transactions on Affective Computing, 2011
- Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic LabelsIEEE Transactions on Affective Computing, 2010
- On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cuesJournal on Multimodal User Interfaces, 2009
- Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysisJournal on Multimodal User Interfaces, 2009
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Facial emotion recognition using multi-modal information2002