On-the-fly Modulation for Balanced Multimodal Learning
Preprint
Abstract
Multimodal learning is expected to boost model performance by integrating information from different modalities. However, its potential is not fully exploited because the widely-used joint training strategy, which has a uniform objective for all modalities, leads to imbalanced and under-optimized uni-modal representations. Specifically, we point out that there often exists modality with more discriminative information, e.g., vision of playing football and sound of blowing wind. They could dominate the joint training process, resulting in other modalities being significantly under-optimized. To alleviate this problem, we first analyze the under-optimized phenomenon from both the feed-forward and the back-propagation stages during optimization. Then, On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies are proposed to modulate the optimization of each modality, by monitoring the discriminative discrepancy between modalities during training. Concretely, OPM weakens the influence of the dominant modality by dropping its feature with dynamical probability in the feed-forward stage, while OGM mitigates its gradient in the back-propagation stage. In experiments, our methods demonstrate considerable improvement across a variety of multimodal tasks. These simple yet effective strategies not only enhance performance in vanilla and task-oriented multimodal models, but also in more complex multimodal tasks, showcasing their effectiveness and flexibility. The source code is available at \url{https://github.com/GeWu-Lab/BML_TPAMI2024}.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Diagnosing and Re-learning for Balanced Multimodal Learning2024
- Positive Sample Propagation along the Audio-Visual Event Line2021
- MMTM: Multimodal Transfer Module for CNN Fusion2020
- What Makes Training Multi-Modal Classification Networks Hard?2020
- Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition2019
- A Simple Baseline for Audio-Visual Scene-Aware Dialog2019
- Audio-Visual Event Localization in Unconstrained Videos2018
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Deep Residual Learning for Image Recognition2016
- Temporal Multimodal Learning in Audiovisual Speech Recognition2016