A Transfer Learning-Based Text-Centric Model for Multimodal Sentiment Analysis
Open Access
- 17 March 2025
- journal article
- Published by Iskender AKKURT in International Journal of Computational and Experimental Science and Engineering
- Vol. 11 (2)
Abstract
Multimodal sentiment analysis (MMSA) is a research method that extracts effective information from heterogeneous modal information. Then, MMSA processes the multimodal data and performs sentiment analysis. Along with big data and machine learning development, multimodal sentiment analysis has become a hot research direction in multimodal learning and natural language processing. Although various feature extraction methods and information fusion methods have been continuously proposed, challenges exist in MMSA research. First, in terms of feature extraction, pre-trained models trained with many data sets can obtain higher quality features, but research on how to use these feature extraction methods to extract the best features is still needed. Currently, the more popular feature fusion methods do not focus on the interaction between multiple modal information and the retention of basic information. To overcome these problems a multimodal sentiment analysis model utilizes text features as core modal features, using video and audio modal features as auxiliary modal features, multimodal feature modality attention mechanism to extract the intrinsic connection between different modalities. The attention mechanism uses the features of video modality and audio modality as the focus and then enhances the text modality with the fusion of video modality and modality. To improve the quality of extracted features, this method chooses the transfer learning training method and uses the pre-trained model for processing. This research uses the CMU-MOSI dataset to test the proposed method. Experimental results show that the performance of the proposed model in emotion score prediction and emotion classification tasks exceeds traditional methods and baseline methods.Keywords
This publication has 33 references indexed in Scilit:
- Occlusion-aware facial expression recognition: A deep learning approachMultimedia Tools and Applications, 2023
- Mmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis2022
- Attention-based Spatio-Temporal Graphic LSTM for EEG Emotion Recognition2021
- A Comprehensive Survey on Transfer LearningProceedings of the IEEE, 2021
- End-to-End Multimodal Emotion Recognition Using Deep Neural NetworksIEEE Journal of Selected Topics in Signal Processing, 2017
- Adversarial Discriminative Domain Adaptation2017
- Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks2017
- Individual differences and their measurement: A review of 100 years of research.Journal of Applied Psychology, 2017
- Deep Residual Learning for Image Recognition2016
- Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challengeSpeech Communication, 2011