VAE-Based Adversarial Multimodal Domain Transfer for Video-Level Sentiment Analysis
Open Access
- 11 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 51315-51324
Abstract
Video-level sentiment analysis is a challenging task and requires systems to obtain discriminative multimodal representations that can capture difference in sentiments across various modalities. However, due to diverse distributions of various modalities and the unified multimodal labels are not always adaptable to unimodal learning, the distance difference between unimodal representations increases, and prevents systems from learning discriminative multimodal representations. In this paper, to obtain more discriminative multimodal representations that can further improve systems’ performance, we propose a VAE-based adversarial multimodal domain transfer (VAE-AMDT) and jointly train it with a multi-attention module to reduce the distance difference between unimodal representations. We first perform variational autoencoder (VAE) to make visual, linguistic and acoustic representations follow a common distribution, and then introduce adversarial training to transfer all unimodal representations to a joint embedding space. As a result, we fuse various modalities on this joint embedding space via the multi-attention module, which consists of self-attention, cross-attention and triple-attention for highlighting important sentimental representations over time and modality. Our method improves F1-score of the state-of-the-art by 3.6% on MOSI and 2.9% on MOSEI datasets, and prove its efficacy in obtaining discriminative multimodal representations for video-level sentiment analysis.Keywords
This publication has 26 references indexed in Scilit:
- Novel Audio Features for Music Emotion RecognitionIEEE Transactions on Affective Computing, 2018
- Feature Selection Based Transfer Subspace Learning for Speech Emotion RecognitionIEEE Transactions on Affective Computing, 2018
- Multimodal Machine Learning: A Survey and TaxonomyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Multi-Objective Based Spatio-Temporal Feature Representation Learning Robust to Expression Intensity Variations for Facial Expression RecognitionIEEE Transactions on Affective Computing, 2017
- Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network2017
- OpenFace: An open source facial behavior analysis toolkit2016
- Learning Grimaces by Watching TV2016
- Deep Face Recognition2015
- Speech Emotion Recognition Using CNN2014
- Domain Adaptation via Transfer Component AnalysisIEEE Transactions on Neural Networks, 2010