FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation
Authors:
Preprint
Abstract
The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON, FedNova and FedDyn resort to an optimization perspective, which requires an auxiliary term or re-weights local updates to calibrate the learning bias or the objective inconsistency. However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions. We thus introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We provide theoretical analysis of the possible benefit from FedSkip and conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedSkip.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Federated Transfer Reinforcement Learning for Autonomous Driving2022
- Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIIEEE Transactions on Knowledge and Data Engineering, 2022
- Federated Learning on Non-IID Data Silos: An Experimental Study2022
- A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and ProtectionIEEE Transactions on Knowledge and Data Engineering, 2021
- Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization2021
- Deep Leakage from Gradients2020
- Federated Learning With Blockchain for Autonomous Vehicles: Analysis and Design ChallengesIEEE Transactions on Communications, 2020
- Machine learning: Trends, perspectives, and prospectsScience, 2015