Abstract
Geo-Distributed Machine Learning (Geo-DML) has been proposed to collaborate geographically dispersed data centers (DCs) and train large scale machine learning (ML) models for various applications. While Geo-DML can achieve excellent performance, it also injects massive data traffic into the Wide Area Networks (WANs) in order to exchange gradients during model training process. Such a huge amount of traffic will not only incur network congestion and prolong the training procedure, but also result in straggler problem when DCs are working in heterogeneous network environments. To alleviate these problems, we propose Self-Adaptive Gradient Quantization (SAGQ) for Geo-DML in this work. In SAGQ, each worker DC adopts specific quantization method based on the heterogeneous and dynamic link bandwidth in order to reduce the commu- nication overhead and balance the communication time among worker DCs. By doing so, SAGQ will speed up the Geo-DML training process without sacrificing the ML model performance. Extensive experiments show that compared with the state-of-the- art techniques, SAGQ reduces the Wall-clock time spent to train an ML model by 1.13× -21.31×. In addition, SAGQ can also improve the model accuracy by0.11%-2.27%over baselines.
Funding Information
  • Natural Science Foundation of Sichuan Province (2022NSFSC0543)
  • Open Research Projects of Zhejiang Lab (2021LC0AB04)
  • National Natural Science Foundation of China (62171085, 62272428, 62001087, U20A20156, 61871097)
  • Natural Science Foundation of Anhui Province (2208085MF167)
  • Natural Science Foundation of Jiangsu Province (BK20221261)
  • Hefei Municipal Natural Science Foundation (2022004)

This publication has 24 references indexed in Scilit: