Balanced Multimodal Learning via On-the-fly Gradient Modulation
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8228-8237
Abstract
Multimodal learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multimodal model outperforms its uni-modal counterpart. Specifically, in this paper we point out that existing multimodal discriminative models, in which uniform objective is designed for all modalities, could remain under-optimized uni-modal representations, caused by another dominated modality in some scenarios, e.g., sound in blowing wind event, vision in drawing picture event, etc. To alleviate this optimization imbalance, we propose on-the-fly gradient modulation to adaptively control the optimization of each modality, via monitoring the discrepancy of their contribution towards the learning objective. Further, an extra Gaussian noise that changes dynamically is introduced to avoid possible generalization drop caused by gradient modulation. As a result, we achieve considerable improvement over common fusion methods on different multimodal tasks, and this simple strategy can also boost existing multimodal methods, which illustrates its efficacy and versatility. The source code is available at https://github.com/GeWu-Lab/OGM-GE_CVPR2022.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition2019
- A comprehensive study on bilingual and multilingual speech emotion recognition using a two-pass classification schemePLOS ONE, 2019
- Deep Multimodal Clustering for Unsupervised Audiovisual Learning2019
- Dense Multimodal Fusion for Hierarchically Joint Representation2019
- Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks2018
- Look, Listen and Learn2017
- Deep Residual Learning for Image Recognition2016
- Temporal Multimodal Learning in Audiovisual Speech Recognition2016
- VQA: Visual Question Answering2015
- CREMA-D: Crowd-Sourced Emotional Multimodal Actors DatasetIEEE Transactions on Affective Computing, 2014