Multimodal Relational Tensor Network for Sentiment and Emotion Classification
Preprint
Abstract
Understanding Affect from video segments has brought researchers from the language, audio and video domains together. Most of the current multimodal research in this area deals with various techniques to fuse the modalities, and mostly treat the segments of a video independently. Motivated by the work of (Zadeh et al., 2017) and (Poria et al., 2017), we present our architecture, Relational Tensor Network, where we use the inter-modal interactions within a segment (intra-segment) and also consider the sequence of segments in a video to model the inter-segment inter-modal interactions. We also generate rich representations of text and audio modalities by leveraging richer audio and linguistic context alongwith fusing fine-grained knowledge based polarity scores from text. We present the results of our model on CMU-MOSEI dataset and show that our model outperforms many baselines and state of the art methods for sentiment classification and emotion recognition.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Advanced LSTM: A Study About Better Time Dependency Modeling in Emotion Recognition2018
- An Ensemble Framework of Voice-Based Emotion Recognition System for Films and TV Programs2018
- Deep Contextualized Word Representations2018
- Context-Dependent Sentiment Analysis in User-Generated Videos2017
- Tensor Fusion Network for Multimodal Sentiment Analysis2017
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media TextProceedings of the International AAAI Conference on Web and Social Media, 2014
- COVAREP — A collaborative voice analysis repository for speech technologies2014
- Class-level spectral features for emotion recognitionSpeech Communication, 2010
- Algorithms on Strings, Trees and Sequences1997