PMR: Prototypical Modal Rebalance for Multimodal Learning
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 20029-20038
Abstract
Multimodal learning (MML) aims to jointly exploit the common priors of different modalities to compensate for their inherent limitations. However, existing MML methods often optimize a uniform objective for different modalities, leading to the notorious “modality imbalance” problem and counterproductive MML performance. To address the problem, some existing methods modulate the learning pace based on the fused modality, which is dominated by the better modality and eventually results in a limited improvement on the worse modal. To better exploit the features of multimodal, we propose Prototypical Modality Rebalance (PMR) to perform stimulation on the particular slow-learning modality without interference from other modalities. Specifically, we introduce the prototypes that represent general features for each class, to build the non-parametric classifiers for uni-modal performance evaluation. Then, we try to accelerate the slow-learning modality by enhancing its clustering toward prototypes. Furthermore, to alleviate the suppression from the dominant modality, we introduce a prototype-based entropy regularization term during the early training stage to prevent premature convergence. Besides, our method only relies on the representations of each modality and without restrictions from model structures and fusion methods, making it with great application potential for various scenarios. The source code is available here 1 1 https://github.com/fanyunfeng-bit/Modal-Imbalance-PMR.Keywords
All Related Versions
Funding Information
- Shenzhen Science and Technology Innovation Commission (JCYJ20200109142008673)
This publication has 32 references indexed in Scilit:
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Deep Multimodal Feature Analysis for Action Recognition in RGB+D VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Revisiting Visual Question Answering Baselines2016
- Deep Residual Learning for Image Recognition2016
- Deep multimodal learning for Audio-Visual Speech Recognition2015
- CREMA-D: Crowd-Sourced Emotional Multimodal Actors DatasetIEEE Transactions on Affective Computing, 2014
- Sample-Specific Late Fusion for Visual Category Recognition2013
- Discriminative model fusion for semantic concept detection and annotation in video2003
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- A Stochastic Approximation MethodThe Annals of Mathematical Statistics, 1951