A Comprehensive Empirical Study of Heterogeneity in Federated Learning
- 7 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 10 (16) , 14071-14083
Abstract
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private data sets owned by nontrusting entities. FL has seen successful deployment in production environments, and it has been adopted in services, such as virtual keyboards, auto-completion, item recommendation, and several IoT applications. However, FL comes with the challenge of performing training over largely heterogeneous data sets, devices, and networks that are out of the control of the centralized FL server. Motivated by this inherent challenge, we aim to empirically characterize the impact of device and behavioral heterogeneity on the trained model. We conduct an extensive empirical study spanning nearly 1.5K unique configurations on five popular FL benchmarks. Our analysis shows that these sources of heterogeneity have a major impact on both model quality and fairness, causing up to $4.6\times $ and $2.2\times $ degradation in the quality and fairness, respectively, thus shedding light on the importance of considering heterogeneity in FL system design.
Keywords
Funding Information
- King Abdullah University of Science and Technology (ORA-CRG10-2021-4699)
This publication has 24 references indexed in Scilit:
- Exploiting Unintended Feature Leakage in Collaborative Learning2019
- Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Practical Secure Aggregation for Privacy-Preserving Machine Learning2017
- Heterogeneity-aware Distributed Parameter Servers2017
- EMNIST: Extending MNIST to handwritten letters2017
- Deep Learning with Differential Privacy2016
- Deep Learning Face Attributes in the Wild2015
- Scaling Distributed Machine Learning with the Parameter Server2014
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009