Towards Optimal Multi-Modal Federated Learning on Non-IID Data with Hierarchical Gradient Blending
Authors:
- 2 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1469-1478
Abstract
Recent advances in federated learning (FL) made it feasible to train a machine learning model across multiple clients, even with non-IID data distributions. In contrast to these uni-modal models that have been studied extensively in the literature, there are few in-depth studies on how multi-modal models can be trained effectively with federated learning. Unfortunately, we empirically observed a counter-intuitive phenomenon that, compared with its uni-modal counterpart, multi-modal FL leads to a significant degradation in performance.Our in-depth analysis of such a phenomenon shows that modality sub-networks and local models can overfit and generalize at different rates. To alleviate these inconsistencies in collaborative learning, we propose hierarchical gradient blending (HGB), which simultaneously computes the optimal blending of modalities and the optimal weighting of local models by adaptively measuring their overfitting and generalization behaviors. When HGB is applied, we present a few important theoretical insights and convergence guarantees for convex and smooth functions, and evaluate its performance in multi-modal FL. Our experimental results on an extensive array of non-IID multi-modal data have demonstrated that HGB is not only able to outperform the best uni-modal baselines but also to achieve superior accuracy and convergence speed as compared to state-of-the-art frameworks.Keywords
This publication has 10 references indexed in Scilit:
- Optimizing Federated Learning on Non-IID Data with Reinforcement Learning2020
- FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding2020
- What Makes Training Multi-Modal Classification Networks Hard?2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020
- FedVision: An Online Visual Object Detection Platform Powered by Federated LearningProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Federated Learning for Vision-and-Language Grounding ProblemsProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Video Classification With Channel-Separated Convolutional Networks2019
- Learning Private Neural Language Modeling with Attentive Aggregation2019
- When Edge Meets Learning: Adaptive Control for Resource-Constrained Distributed Machine Learning2018
- Multimodal Machine Learning: A Survey and TaxonomyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018