ESync: Accelerating Intra-Domain Federated Learning in Heterogeneous Data Centers
- 11 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Services Computing
- Vol. 15 (4) , 2261-2274
Abstract
Federated Learning (FL) serves privacy-preserving collaborative learning among multiple isolated parties, while retaining their privacy data locally. Cross-device and cross-silo FL have achieved great success in cross-domain applications, in which the scarce communication resource is the primary bottleneck. Driven by the need to combine heterogeneous machines from different parties to build a shared data center, we found intra-domain FL, a new type of FL in which isolated parties collaborate in the shared data center, and strong computational heterogeneity becomes the primary bottleneck. To mitigate the training inefficiency caused by stragglers, this paper proposes an efficient synchronization algorithm ESync, which allows parties to train different iterations locally under the coordination of a novel scheduler State Server. We give the boundaries of weight divergence and optimality gap of ESync, and analyze the trade-off between convergence accuracy and communication efficiency. Extensive experiments are conducted to compare ESync with SSGD, ASGD, DC-ASGD, FedAvg, FedAsync, TiFL and FedDrop under strong computational heterogeneity. Numerical results show that ESync achieves great speed up without loss of accuracy, and therefore demonstrate the effectiveness of ESync in both training efficiency and converged accuracy.Keywords
Funding Information
- National Key Research and Development Program of China (2019YFB1802800)
- PCL Future Greater-Bay Area Network Facilities for Large-scale Experiments and Applications (LZC0019)
This publication has 11 references indexed in Scilit:
- Advances and Open Problems in Federated LearningFoundations and Trends in Machine Learning, 2021
- TiFL: A Tier-based Federated Learning System2020
- BOA: batch orchestration algorithm for straggler mitigation of distributed DL training in heterogeneous GPU clusterThe Journal of Supercomputing, 2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Anytime Exploitation of Stragglers in Synchronous Stochastic Gradient Descent2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- The EU General Data Protection Regulation (GDPR)2017
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Communication-efficient algorithms for statistical optimization2012