Aperiodic Local SGD: Beyond Local SGD
Authors:
Open Access
- 29 August 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Abstract
Variations of stochastic gradient decedent (SGD) methods are at the core of training deep neural network models. However, in distributed deep learning, where multiple computing devices and data segments are employed in the training process, the performance of SGD can be significantly limited by the overhead of gradient communication. Local SGD methods are designed to overcome this bottleneck by averaging individual gradients trained over parallel workers after multiple local iterations. Currently, both for theoretical analyses and for practical applications, most studies employ periodic synchronization scheme by default, while few of them focus on the aperiodic schemes to obtain better performance models with limited computation and communication overhead. In this paper, we investigate local SGD with an arbitrary synchronization scheme to answer two questions: (1) Is the periodic synchronization scheme best? (2) If not, what is the optimal one? First, for any synchronization scheme, we derive the performance boundary with fixed overhead, and formulate the performance optimization under given computation and communication constraints. Then we find a succinct property of the optimal scheme that the local iteration number decreases as training continues, which indicates the periodic one is suboptimal. Furthermore, with some reasonable approximations, we obtain an explicit form of the optimal scheme and propose Aperiodic Local SGD (ALSGD) as an improved substitute for local SGD without any overhead increment. Our experiments also confirm that with the same computation and communication overhead, ALSGD outperforms local SGD in performance, especially for heterogeneous data.Keywords
Funding Information
- National Natural Science Foundation of Heilongjiang Province (YQ2019F007)
- Programs for Science and Technology Development of Heilongjiang Province (2021ZXJ05A03)
- Key Science Technology Specific Projects of Heilongjiang Province (2019ZX14A01)
- National Key R&D Program of China (2021ZD0110905)
- National Natural Science Foundation of China (62106061, 61972114, 62072137)
- Fundamental Research Funds for the Central Universities (AUGA5710010521)
This publication has 7 references indexed in Scilit:
- GradientFlow: Optimizing Network Performance for Large-Scale Distributed DNN TrainingIEEE Transactions on Big Data, 2019
- Demystifying Parallel and Distributed Deep LearningACM Computing Surveys, 2019
- Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent2019
- Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep LearningProceedings of the AAAI Conference on Artificial Intelligence, 2019
- MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms2019
- Group Normalization2018
- On the Convergence Properties of a K-step Averaging Stochastic Gradient Descent Algorithm for Nonconvex Optimization2018