FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation
Authors:
- 1 November 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 131-140
Abstract
The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON, FedNova and FedDyn resort to an optimization perspective, which requires an auxiliary term or re-weights local updates to calibrate the learning bias or the objective inconsistency. However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions. We thus introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We provide theoretical analysis of the possible benefit from FedSkip and conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiencys. Source code is available at: https://github.com/MediaBrain-SJTU/FedSkip.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Device-Cloud Collaborative Learning for Recommendation2021
- Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization2021
- Deep Leakage from Gradients2020
- Federated Visual Classification with Real-World Data Distribution2020
- Federated Learning With Blockchain for Autonomous Vehicles: Analysis and Design ChallengesIEEE Transactions on Communications, 2020
- FedVision: An Online Visual Object Detection Platform Powered by Federated LearningProceedings of the AAAI Conference on Artificial Intelligence, 2020
- A Survey on Distributed Machine LearningACM Computing Surveys, 2020
- EMNIST: Extending MNIST to handwritten letters2017
- Privacy-Preserving Deep Learning2015
- Machine learning: Trends, perspectives, and prospectsScience, 2015