Multimodal Federated Learning: A Survey
Preprint
- 20 July 2023
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Federated learning (FL) has become a burgeoning and attractive research area, which provides a collaborative training scheme for distributed data sources with privacy concerns. Most existing FL studies focus on taking unimodal data, such as images and text, as the model input and resolving the heterogeneity challenge, i.e., the non-identically distributed (non-IID) challenge caused by data distribution imbalance related to data labels and data amount. In real-world applications, data are usually described by multiple modalities. However, to the best of our knowledge, only a handful of studies have been proposed to improve the system performance by utilizing multimodal data. In this survey paper, we identify the significance of this emerging research topic – multimodal federated learning (MFL) and perform a literature review on the state-of-art MFL methods. Furthermore, we categorize multi-modal federated learning into congruent and incongruent multimodal federated learning based on whether all clients possess the same modal combinations. We investigate the feasible application tasks and related benchmarks for MFL. Lastly, we summarize the promising directions and fundamental challenges in this field for future research.Keywords
All Related Versions
- Version 1, 2023-07-20, Preprints
- Published version: Sensors, 23 (15), 6986.
This publication has 60 references indexed in Scilit:
- MIMIC-IV, a freely accessible electronic health record datasetScientific Data, 2023
- Reducing variations in multi-center Alzheimer’s disease classification with convolutional adversarial autoencoderMedical Image Analysis, 2022
- A unified framework for multi-modal federated learningNeurocomputing, 2022
- FedTriNet: A Pseudo Labeling Method with Three Players for Federated Semi-supervised Learning2021
- Federated learning on non-IID data: A surveyNeurocomputing, 2021
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100International Journal of Computer Vision, 2021
- The future of digital health with federated learningnpj Digital Medicine, 2020
- Deep Multimodal Representation Learning: A SurveyIEEE Access, 2019
- 3D ShapeNets: A deep representation for volumetric shapes2015
- Vehicle classification in distributed sensor networksJournal of Parallel and Distributed Computing, 2004