FedGKD: Toward Heterogeneous Federated Learning via Global Knowledge Distillation
- 15 September 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computers
- Vol. 73 (1) , 3-17
Abstract
Federated learning, as one enabling technology of edge intelligence, has gained substantial attention due to its efficacy in training deep learning models without data privacy and network bandwidth concerns. However, due to the heterogeneity of the edge computing system and data, many methods suffer from the “ client-drift ” issue that could considerably impede the convergence of global model training: local models on clients can drift apart, and the aggregated model can be different from the global optimum. To tackle this issue, one intuitive idea is to guide the local model training by global teachers, i.e. , past global models, where each client learns the global knowledge from past global models via adaptive knowledge distillation techniques. Inspired by these insights, we propose a novel approach for heterogeneous federated learning, F ed GKD, which fuses the knowledge from historical global models and guides local training to alleviate the “ client-drift ” issue. In this paper, we evaluate F ed GKD through extensive experiments across various CV and NLP datasets (i.e., CIFAR-10/100, Tiny-ImageNet, AG News, SST5) under different heterogeneous settings. The proposed method is guaranteed to converge under common assumptions and outperforms the state-of-the-art baselines in the non-IID federated setting.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFB1714600)
- National Natural Science Foundation of China (62072204, 62032008, 62102157)
- National Science Foundation (CRII-2246067)
This publication has 29 references indexed in Scilit:
- SAFA: A Semi-Asynchronous Protocol for Fast Federated Learning With Low OverheadIEEE Transactions on Computers, 2021
- LDP-Fed2020
- A Neural Network-Based On-device Learning Anomaly Detector for Edge DevicesIEEE Transactions on Computers, 2020
- Robust and Communication-Efficient Federated Learning From Non-i.i.d. DataIEEE Transactions on Neural Networks and Learning Systems, 2019
- Wireless Edge Computing With Latency and Reliability GuaranteesProceedings of the IEEE, 2019
- Deep Mutual Learning2018
- Network Topology and Communication-Computation Tradeoffs in Decentralized OptimizationProceedings of the IEEE, 2018
- Practical Secure Aggregation for Privacy-Preserving Machine Learning2017
- Deep Residual Learning for Image Recognition2016
- Wiley Series in Probability and Statistics2008