Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning
Preprint
Abstract
Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. Recently, the use of small pre-trained models has been shown effective in federated learning optimization and improving convergence. However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters. In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed. Can we find a solution to enable those strong and readily-available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning and thus introduce a new framework: FedPEFT. Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive or even better performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Current trends in deep learning for Earth Observation: An open-source benchmark arena for image classificationISPRS Journal of Photogrammetry and Remote Sensing, 2023
- AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimation2022
- Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning2022
- GradViT: Gradient Inversion of Vision Transformers2022
- Grounded Language-Image Pre-training2022
- Emerging Properties in Self-Supervised Vision Transformers2021
- Advances and Open Problems in Federated Learning2021
- Local Differential Privacy for Deep LearningIEEE Internet of Things Journal, 2019
- Remote Sensing Image Scene Classification: Benchmark and State of the ArtProceedings of the IEEE, 2017
- The Algorithmic Foundations of Differential Privacy2013