GRACE: A Compressed Communication Framework for Distributed Machine Learning
Authors:
- 1 July 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 561-572
Abstract
Powerful computer clusters are used nowadays to train complex deep neural networks (DNN) on large datasets. Distributed training increasingly becomes communication bound. For this reason, many lossy compression techniques have been proposed to reduce the volume of transferred data. Unfortunately, it is difficult to argue about the behavior of compression methods, because existing work relies on inconsistent evaluation testbeds and largely ignores the performance impact of practical system configurations. In this paper, we present a comprehensive survey of the most influential compressed communication methods for DNN training, together with an intuitive classification (i.e., quantization, sparsification, hybrid and low-rank). Next, we propose GRACE, a unified framework and API that allows for consistent and easy implementation of compressed communication on popular machine learning toolkits. We instantiate GRACE on TensorFlow and PyTorch, and implement 16 such methods. Finally, we present a thorough quantitative evaluation with a variety of DNNs (convolutional and recurrent), datasets and system configurations. We show that the DNN architecture affects the relative performance among methods. Interestingly, depending on the underlying communication library and computational cost of compression / decompression, we demonstrate that some methods may be impractical. GRACE and the entire benchmarking suite are available as open-source.Keywords
This publication has 17 references indexed in Scilit:
- In-Network Computation is a Dumb Idea Whose Time Has Come2017
- Densely Connected Convolutional Networks2017
- Sparse Communication for Distributed Gradient Descent2017
- Communication Quantization for Data-Parallel Training of Deep Neural Networks2016
- Gradient methods for minimizing composite functionsMathematical Programming, 2012
- ImageNet: A large-scale hierarchical image database2009
- Space-efficient online computation of quantile summaries2001
- Long Short-Term MemoryNeural Computation, 1997
- Simplified neuron model as a principal component analyzerJournal of Mathematical Biology, 1982
- A Stochastic Approximation MethodThe Annals of Mathematical Statistics, 1951