Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data
Preprint
Abstract
SGD with momentum is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum is Distributed SGD (DSGD) with momentum (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several studies have addressed this issue and proposed methods with momentum that are more robust to data heterogeneity than DSGDm, although their convergence rates remain dependent on data heterogeneity and deteriorate when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which is a method with momentum whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the convergence rate of Momentum Tracking in the setting where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient $\beta \in [0, 1)$. Through experiments, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum and can consistently outperform these existing methods when the data distributions are heterogeneous.
Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Communication Compression for Decentralized Learning With Operator Splitting MethodsIEEE Transactions on Signal and Information Processing over Networks, 2023
- GTAdam: Gradient Tracking With Adaptive Momentum for Distributed Online OptimizationIEEE Transactions on Control of Network Systems, 2022
- Fast Decentralized Nonconvex Finite-Sum Optimization with Recursive Variance ReductionSIAM Journal on Optimization, 2022
- Distributed stochastic gradient tracking methodsMathematical Programming, 2020
- Distributed Heavy-Ball: A Generalization and Acceleration of First-Order Methods With Gradient TrackingIEEE Transactions on Automatic Control, 2019
- Achieving Geometric Convergence for Distributed Optimization Over Time-Varying GraphsSIAM Journal on Optimization, 2017
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998