Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10164-10173
Abstract
Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. Data heterogeneity is one of the main challenges in FL, which results in slow convergence and degraded performance. Most existing approaches only tackle the heterogeneity challenge by restricting the local model update in client, ignoring the performance drop caused by direct global model aggregation. Instead, we propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG), which relieves the issue of direct model aggregation. Concretely, FedFTG explores the input space of local models through a generator, and uses it to transfer the knowledge from local models to the global model. Besides, we propose a hard sample mining scheme to achieve effective knowledge distillation throughout the training. In addition, we develop customized label sampling and class-level ensemble to derive maximum utilization of knowledge, which implicitly mitigates the distribution discrepancy across clients. Extensive experiments show that our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.Keywords
Funding Information
- National Natural Science Foundation of China (62088102)
This publication has 11 references indexed in Scilit:
- FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space2021
- Model-Contrastive Federated Learning2021
- Particle Swarm Optimized Federated Learning For Industrial IoT and Smart City Services2020
- Federated Learning in Smart City Sensing: Challenges and OpportunitiesSensors, 2020
- Performance Optimization of Federated Person Re-identification via Benchmark Analysis2020
- Communication-Efficient Federated Learning for Wireless Edge Intelligence in IoTIEEE Internet of Things Journal, 2019
- Data-Free Learning of Student Networks2019
- MIO-TCD: A New Benchmark Dataset for Vehicle Classification and LocalizationIEEE Transactions on Image Processing, 2018
- Deep Residual Learning for Image Recognition2016
- A large-scale car dataset for fine-grained categorization and verification2015