Multi-Task Supervised Alignment Pre-Training for Few-Shot Multimodal Sentiment Analysis
Open Access
- 17 February 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (4) , 2095
Abstract
Few-shot multimodal sentiment analysis (FMSA) has garnered substantial attention due to the proliferation of multimedia applications, especially given the frequent difficulty in obtaining large quantities of training samples. Previous works have directly incorporated vision modality into the pre-trained language model (PLM) and then leveraged prompt learning, showing effectiveness in few-shot scenarios. However, these methods encounter challenges in aligning the high-level semantics of different modalities due to their inherent heterogeneity, which impacts the performance of sentiment analysis. In this paper, we propose a novel framework called Multi-task Supervised Alignment Pre-training (MSAP) to enhance modality alignment and consequently improve the performance of multimodal sentiment analysis. Our approach uses a multi-task training method—incorporating image classification, image style recognition, and image captioning—to extract modal-shared information and stronger semantics to improve visual representation. We employ task-specific prompts to unify these diverse objectives into a single Masked Language Model (MLM), which serves as the foundation for our Multi-task Supervised Alignment Pre-training (MSAP) framework to enhance the alignment of visual and textual modalities. Extensive experiments on three datasets demonstrate that our method achieves a new state-of-the-art for the FMSA task.Keywords
This publication has 25 references indexed in Scilit:
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- A Deep Multi-level Attentive Network for Multimodal Sentiment AnalysisACM Transactions on Multimedia Computing, Communications, and Applications, 2023
- AOBERT: All-modalities-in-One BERT for multimodal sentiment analysisInformation Fusion, 2022
- Heterogeneous graph convolution based on In-domain Self-supervision for Multimodal Sentiment AnalysisExpert Systems with Applications, 2022
- Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directionsInformation Fusion, 2022
- Multimodal Sentiment Analysis With Two-Phase Multi-Task LearningIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment AnalysisIEEE Transactions on Affective Computing, 2022
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- Image-Text Multimodal Emotion Classification via Multi-View Attentional NetworkIEEE Transactions on Multimedia, 2020
- A Deep Multi-task Contextual Attention Framework for Multi-modal Affect AnalysisACM Transactions on Knowledge Discovery From Data, 2020