Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
Authors:
- 1 November 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1207-1208
Abstract
The increasing size of machine learning models, especially deep neural network models, can improve the model generalization capability. However, large models require more training data and more computing resources (such as GPU clusters) to train. In distributed training, the communication overhead of exchanging gradients or models among workers becomes a potential system bottleneck that limits the system scalability. Recently, many research works aim to reduce communication time of two types of distributed deep learning architectures, centralized and decentralized.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs2020
- A Convergence Analysis of Distributed SGD with Communication-Efficient Gradient Sparsification2019
- A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks2019
- Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs2018