Self-Adaptive Gradient Quantization for Geo-Distributed Machine Learning Over Heterogeneous and Dynamic Networks
- 5 July 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 11 (4) , 3483-3496
Abstract
Geo-Distributed Machine Learning (Geo-DML) has been proposed to collaborate geographically dispersed data centers (DCs) and train large scale machine learning (ML) models for various applications. While Geo-DML can achieve excellent performance, it also injects massive data traffic into the Wide Area Networks (WANs) in order to exchange gradients during model training process. Such a huge amount of traffic will not only incur network congestion and prolong the training procedure, but also result in straggler problem when DCs are working in heterogeneous network environments. To alleviate these problems, we propose Self-Adaptive Gradient Quantization (SAGQ) for Geo-DML in this work. In SAGQ, each worker DC adopts specific quantization method based on the heterogeneous and dynamic link bandwidth in order to reduce the commu- nication overhead and balance the communication time among worker DCs. By doing so, SAGQ will speed up the Geo-DML training process without sacrificing the ML model performance. Extensive experiments show that compared with the state-of-the- art techniques, SAGQ reduces the Wall-clock time spent to train an ML model by 1.13× -21.31×. In addition, SAGQ can also improve the model accuracy by0.11%-2.27%over baselines.Keywords
Funding Information
- Natural Science Foundation of Sichuan Province (2022NSFSC0543)
- Open Research Projects of Zhejiang Lab (2021LC0AB04)
- National Natural Science Foundation of China (62171085, 62272428, 62001087, U20A20156, 61871097)
- Natural Science Foundation of Anhui Province (2208085MF167)
- Natural Science Foundation of Jiangsu Province (BK20221261)
- Hefei Municipal Natural Science Foundation (2022004)
This publication has 24 references indexed in Scilit:
- Optimizing Federated Learning on Non-IID Data with Reinforcement Learning2020
- Accelerating Distributed Deep Learning By Adaptive Gradient Quantization2020
- PipeDream2019
- MQGrad2018
- Deep Residual Learning for Image Recognition2016
- Convex Optimization: Algorithms and ComplexityFoundations and Trends in Machine Learning, 2015
- Low Latency Geo-distributed Data AnalyticsACM SIGCOMM Computer Communication Review, 2015
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic ProgrammingSIAM Journal on Optimization, 2013
- JitterPath: Probing Noise Resilient One-Way Delay Jitter-Based Available Bandwidth EstimationIEEE Transactions on Multimedia, 2007
- Evaluation and characterization of available bandwidth probing techniquesIEEE Journal on Selected Areas in Communications, 2003