Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
Preprint
Abstract
Sensory training during the early ages is vital for human development. Inspired by this cognitive phenomenon, we observe that the early training stage is also important for the multimodal learning process, where dataset information is rapidly acquired. We refer to this stage as the prime learning window. However, based on our observation, this prime learning window in multimodal learning is often dominated by information-sufficient modalities, which in turn suppresses the information acquisition of information-insufficient modalities. To address this issue, we propose Information Acquisition Regulation (InfoReg), a method designed to balance information acquisition among modalities. Specifically, InfoReg slows down the information acquisition process of information-sufficient modalities during the prime learning window, which could promote information acquisition of information-insufficient modalities. This regulation enables a more balanced learning process and improves the overall performance of the multimodal network. Experiments show that InfoReg outperforms related multimodal imbalanced methods across various datasets, achieving superior model performance. The code is available at https://github.com/GeWu-Lab/InfoReg_CVPR2025.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Diagnosing and Re-learning for Balanced Multimodal Learning2024
- Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open QuestionsACM Computing Surveys, 2024
- Seizing Critical Learning Periods in Federated LearningProceedings of the AAAI Conference on Artificial Intelligence, 2022
- Learning to Balance the Learning Rates Between Various Modalities via Adaptive Tracking FactorIEEE Signal Processing Letters, 2021
- Emergence of Invariance and Disentanglement in Deep Representations2018