MASANet: Multi-Aspect Semantic Auxiliary Network for Visual Sentiment Analysis
- 4 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 15 (3) , 1439-1450
Abstract
Recently, multi-modal affective computing has demonstrated that introducing multi-modal information can enhance performance. However, multi-modal research faces significant challenges due to its high requirements regarding data acquisition, modal integrity, and feature alignment. The widespread use of multi-modal pre-training methods offers the possibility of aiding visual sentiment analysis by introducing cross-domain knowledge. This paper proposes a Multi-Aspect Semantic Auxiliary Network (MASANet) for visual sentiment analysis. Specifically, MASANet achieves modality expansion through cross-modal generation, making it possible to introduce cross-domain semantic assistance. Then, a cross-modal gating module and an adaptive modal fusion module are proposed for aspect-level and cross-modal interaction, respectively. In addition, a designed semantic polarity constraint loss is presented to improve sentiment multi-classification performance. Evaluations of eight widely-used affective image datasets demonstrate that our proposed method outperforms the state-of-the-art methods. Further ablation experiments and visualization results also confirm the effectiveness of the proposed method and its modules.Keywords
Funding Information
- National Natural Science Foundation of China (61972163, U1801262)
- Natural Science Foundation of Guangdong Province (2022A1515011555, 2023A1515012568)
- Guangdong Provincial Key Laboratory of Human Digital Twin (2022B1212010004)
- Key Laboratory of Cognitive Radio and Information Processing
- Ministry of Education, Guilin University of Electronic Technology
- Pazhou Lab
This publication has 57 references indexed in Scilit:
- A novel dropout mechanism with label extension schema toward text emotion classificationInformation Processing & Management, 2022
- MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis2022
- Exploiting emotional concepts for image emotion recognitionThe Visual Computer, 2022
- ArtEmis: Affective Language for Visual Art2021
- A Circular-Structured Representation for Visual Emotion Distribution Learning2021
- Object semantics sentiment correlation analysis enhanced image sentiment classificationKnowledge-Based Systems, 2020
- Learning Multi-level Deep Representations for Image Emotion ClassificationNeural Processing Letters, 2019
- Multi-level region-based Convolutional Neural Network for image emotion classificationNeurocomputing, 2019
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Deep Residual Learning for Image Recognition2016