Hierarchical Multi-Criteria Representation Fusion for Robust Incomplete Multimodal Sentiment Analysis
- 23 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 17 (1) , 290-302
Abstract
The challenge of improving robustness to missing data in multimodal sentiment analysis (MSA) has recently attracted increasing attention. The existing work is based on sentence-level sentiment analysis, which requires extracting vector representations from input multimodal sentence sequences to perform sentiment regression or classification tasks. However, most studies focus primarily on exploring cross-modal correlations, neglecting the dynamic semantic variations across sequence frames during feature merging and fusion. In this paper, we propose the Hierarchical Multi-criteria Representation Fusion (HMRF) framework, which effectively captures dynamic semantic information across frames and modalities. Specifically, HMRF consists of two parts, i.e., sequence merging and information transfer. For sequence merging, we design a hierarchical cross-modal semantic perception mechanism and a multi-criteria feature merging mechanism, which together mitigate unimodal affective bias and progressively integrate incomplete feature sequences. Moreover, for information transfer, hierarchical information distillation and high-level semantic reconstruction are performed to transfer certain attributes and sentiment information from the complete view to the incomplete view during training. Extensive experiments on multiple benchmark datasets consistently demonstrate that the proposed HMRF outperforms existing baselines.Keywords
Funding Information
- National Natural Science Foundation of China (62572145)
- National Natural Science Foundation of China (62206073)
- National Natural Science Foundation of China (62176077)
- National Natural Science Foundation of China (62272133)
- Shenzhen Key Technical Project (JCYJ20241202123728037)
- Shenzhen Key Technical Project (JSGG20220831092805009)
- Shenzhen Key Technical Project (JSGG20220831105603006)
- Shenzhen Key Technical Project (KJZD20230923115117033)
- Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies (2022B1212010005)
- Shenzhen Science and Technology Program (KJZD20230923114600002)
- Shenzhen Colleges and Universities Stable Support Program (GXWD20220811170100001)
This publication has 39 references indexed in Scilit:
- Multi-Criteria Token Fusion with One-Step-Ahead Attention for Efficient Vision Transformers2024
- Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities2024
- Distribution-Consistent Modal Recovering for Incomplete Multimodal Learning2023
- Exploiting Modality-Invariant Feature for Robust Multimodal Emotion Recognition with Missing Modalities2023
- Decoupled Multimodal Distilling for Emotion Recognition2023
- Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis ResearchIEEE Transactions on Affective Computing, 2023
- TETFN: A text enhanced transformer fusion network for multimodal sentiment analysisPattern Recognition, 2022
- OpenFace 2.0: Facial Behavior Analysis Toolkit2018
- Multimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal MessagesIEEE Intelligent Systems, 2016
- COVAREP — A collaborative voice analysis repository for speech technologies2014