Adaptive Modality Distillation for Separable Multimodal Sentiment Analysis
- 9 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Intelligent Systems
- Vol. 36 (3) , 82-89
Abstract
Multimodal sentiment analysis has increasingly attracted attention since with the arrival of complementary data streams, it has great potential to improve and go beyond unimodal sentiment analysis. In this article, we present an efficient separable multimodal learning method to deal with the tasks with modality missing issue. In this method, the multimodal tensor is utilized to guide the evolution of each separated modality representation. To save the computational expense, Tucker decomposition is introduced, which leads to a general extension of the low-rank tensor fusion method with more modality interactions. The method, in turn, enhances our modality distillation processing. Comprehensive experiments on three popular multimodal sentiment analysis datasets, CMU-MOSI, POM, and IEMOCAP, show a superior performance especially when only partial modalities are available.Keywords
Funding Information
- Academy of Finland for ICT (328,115)
- Medical Insurance Group of Australia (316,765)
- Infotech Oulu
- National Key Research and Development Program of China (2019YFB1312000)
- National Natural Science Foundation of China (62076195)
This publication has 16 references indexed in Scilit:
- Sparse Tikhonov-Regularized Hashing for Multi-Modal Learning2018
- MUTAN: Multimodal Tucker Fusion for Visual Question Answering2017
- Affective Computing and Sentiment Analysis2017
- Tensor Fusion Network for Multimodal Sentiment Analysis2017
- Deep multimodal fusion for persuasiveness prediction2016
- Learning with Side Information through Modality Hallucination2016
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding2016
- Computational Analysis of Persuasiveness in Social Multimedia2014
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Support-vector networksMachine Learning, 1995