Low Rank Fusion based Transformers for Multimodal Sequences
Preprint
Abstract
Our senses individually work in a coordinated fashion to express our emotional intentions. In this work, we experiment with modeling modality-specific sensory signals to attend to our latent multimodal emotional intentions and vice versa expressed via low-rank multimodal fusion and multimodal transformers. The low-rank factorization of multimodal fusion amongst the modalities helps represent approximate multiplicative latent signal interactions. Motivated by the work of~\cite{tsai2019MULT} and~\cite{Liu_2018}, we present our transformer-based cross-fusion architecture without any over-parameterization of the model. The low-rank fusion helps represent the latent signal interactions while the modality-specific attention helps focus on relevant parts of the signal. We present two methods for the Multimodal Sentiment and Emotion Recognition results on CMU-MOSEI, CMU-MOSI, and IEMOCAP datasets and show that our models have lesser parameters, train faster and perform comparably to many larger fusion-based architectures.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization2019
- Efficient Low-rank Multimodal Fusion With Modality-Specific Factors2018
- Tensor Fusion Network for Multimodal Sentiment Analysis2017
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- IEMOCAP: interactive emotional dyadic motion capture databaseLanguage Resources and Evaluation, 2008
- Long Short-Term MemoryNeural Computation, 1997