Image–Text Multimodal Sentiment Analysis with Large Model-Generated Descriptive Semantics and Difference-Aware Gated Fusion
Open Access
- 15 August 2026
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 15 (16) , 3639
Abstract
Image–text multimodal sentiment analysis aims to integrate textual and visual information to comprehensively understand sentiment expressions in complex scenarios. However, existing methods focus on cross-modal feature interaction and fusion, and still have difficulty capturing effective sentiment cues in scenarios involving insufficient textual semantics, implicit visual affective cues, and inconsistent sentiment expressions between text and image. To address these issues, this paper proposes an image–text multimodal sentiment analysis method with large model-generated descriptive semantics and difference-aware gated fusion. Specifically, a large model generates semantic descriptions for image–text pairs, from which an enhanced semantic view is constructed to supplement implicit or insufficiently expressed sentiment cues in the original modalities. An original-enhanced dual-branch structure models the original image–text evidence and enhanced semantic evidence separately. To improve semantic consistency between the two branches, a cross-branch semantic alignment mechanism is introduced to reduce semantic shifts caused by enhanced information. In the fusion stage, difference-aware gated fusion and residual compensation are employed to adaptively balance branch contributions while preserving discriminative branch differences. Experimental results on the MVSA-Single and MVSA-Multiple datasets show that the proposed method improves performance in image–text multimodal sentiment classification, thereby validating the effectiveness of combining semantic enhancement with difference-aware modeling.Keywords
Funding Information
- Xinjiang Uygur Autonomous Region Innovation Environment (Talent and Base) Construction Special Program—Natural Science Program (2025D03032)
- Undergraduate Education and Teaching Research and Reform Project of Universities in the Xinjiang Uygur Autonomous Region (XJGXPTJG-2026077)
- Xinjiang Normal University (XJNUQB2022-22)
- Xinjiang Normal University (XJNU-ZHJY202403)
- National Natural Science Foundation of China (61662081)
- National Social Science Fund of China (14AZD11)
This publication has 34 references indexed in Scilit:
- SemCap: Sentiment-aware semantic captioning for multimodal aspect-based sentiment analysisJournal of King Saud University Computer and Information Sciences, 2026
- Systematic review of recent advances in multimodal sentiment analysisDiscover Computing, 2025
- Resolving sentiment discrepancy for multimodal sentiment detection via semantics completion and decompositionPattern Recognition, 2025
- Large language models meet text-centric multimodal sentiment analysis: a surveyScience China Information Sciences, 2025
- Multimodal large language model enhancement network for multimodal sentiment analysisMultimedia Systems, 2025
- Multi-grained feature gating fusion network for multimodal sentiment analysisKnowledge and Information Systems, 2025
- An Image-Text Sentiment Analysis Method Using Multi-Channel Multi-Modal Joint LearningApplied Artificial Intelligence, 2024
- TETFN: A text enhanced transformer fusion network for multimodal sentiment analysisPattern Recognition, 2022
- A multi-stage multimodal framework for sentiment analysis of Assamese in low resource settingExpert Systems with Applications, 2022
- Multimodal Sentiment Analysis With Image-Text Interaction NetworkIEEE Transactions on Multimedia, 2022