Monte Carlo Gradient Quantization
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3087-3095
Abstract
We propose Monte Carlo methods to leverage both sparsity and quantization to compress gradients of neural networks throughout training. On top of reducing the communication exchanged between multiple workers in a distributed setting, we also improve the computational efficiency of each worker. Our method, called Monte Carlo Gradient Quantization (MCGQ), shows faster convergence and higher performance than existing quantization methods on image classification and language modeling. Using both low-bit-width-quantization and high sparsity levels, our method more than doubles the rates of existing compression methods from 200× to 520× and 462× to more than 1200× on different language modeling tasks.Keywords
This publication has 8 references indexed in Scilit:
- Sparse Communication for Distributed Gradient Descent2017
- Communication Quantization for Data-Parallel Training of Deep Neural Networks2016
- Optimal vector quantization in terms of Wasserstein distanceJournal of Multivariate Analysis, 2011
- Using MPI1999
- Building a Large Annotated Corpus of English: The Penn Treebank1993
- Calculation of the Wasserstein Distance Between Probability Distributions on the LineTheory of Probability & Its Applications, 1974
- A Stochastic Approximation MethodThe Annals of Mathematical Statistics, 1951
- The Monte Carlo MethodJournal of the American Statistical Association, 1949