SIM: Accelerating Distributed DNN Training by Exploring Gradient Similarity
- 26 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Networking
- Vol. 34, 1838-1850
Abstract
Synchronous stochastic gradient descent (SSGD) has been widely used in distributed deep learning. However, since the local gradients need to be shared among workers at every iteration, SSGD performance is significantly influenced by network bottlenecks caused by either heterogeneous environment or bandwidth contention. To solve this problem, asynchronous parallel (ASP) strategy allows each worker to update parameters independently without synchronization, while suffering from accuracy loss and convergence inefficiency. In this paper, we propose a novel similarity-based synchronization scheme called SIM, which mitigates the impact of network bottlenecks and ensures convergence efficiency. Specifically, SIM reduces the number of aggregation workers based on the gradient similarity between global and local gradients, therefore shrinking the waiting time for the stragglers. We provide a theoretical analysis of convergence efficiency and conduct large-scale testbed experiments on CIFAR-10 and SQUAD dataset. The experimental results show that SIM reduces the convergence time of four classical deep learning models by up to 40%.Keywords
Funding Information
- National Natural Science Foundation of China (U24A20245)
- National Natural Science Foundation of China (62132022)
- National Natural Science Foundation of China (U23A20313)
- science and technology innovation Program of Hunan Province (2024RC1005)
This publication has 18 references indexed in Scilit:
- Addressing Network Bottlenecks with Divide-and-Shuffle Synchronization for Distributed DNN Training2022
- Federated Learning on Non-IID Data Silos: An Experimental Study2022
- Live Gradient Compensation for Evading Stragglers in Distributed Learning2021
- GSSP: Eliminating Stragglers Through Grouping Synchronous for Distributed Deep Learning in Heterogeneous ClusterIEEE Transactions on Cloud Computing, 2021
- Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs2020
- Adaptive Distributed Stochastic Gradient Descent for Minimizing Delay in the Presence of Stragglers2020
- DeepCaps: Going Deeper With Capsule Networks2019
- Round-Robin Synchronization: Mitigating Communication Bottlenecks in Parameter Servers2019
- Exploring the Limits of Weakly Supervised Pretraining2018
- Deep Residual Learning for Image Recognition2016