On parallelizability of stochastic gradient descent for speech DNNS
Authors:
- 1 May 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15206149,p. 235-239
Abstract
This paper compares the theoretical efficiency of model-parallel and data-parallel distributed stochastic gradient descent training of DNNs. For a typical Switchboard DNN with 46M parameters, the results are not pretty: With modern GPUs and interconnects, model parallelism is optimal with only 3 GPUs in a single server, while data parallelism with a minibatch size of 1024 does not even scale to 2 GPUs. We further show that data-parallel training efficiency can be improved by increasing the minibatch size (through a combination of AdaGrad and automatic adjustments of learning rate and minibatch size) and data compression. We arrive at an estimated possible end-to-end speed-up of 5 times or more. We do not address issues of robustness to process failure or other issues that might occur during training, nor of speed of convergence differences between ASGD and SGD parameter update patterns.Keywords
This publication has 13 references indexed in Scilit:
- Optimization Techniques to Improve Training Speed of Deep Neural Networks for Large Speech TasksIEEE Transactions on Audio, Speech, and Language Processing, 2013
- A cluster-based multiple deep neural networks method for large vocabulary continuous speech recognition2013
- Asynchronous stochastic gradient descent for DNN training2013
- An empirical study of learning rates in deep neural networks for speech recognition2013
- Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription2011
- Distributed Optimization and Statistical Learning via the Alternating Direction Method of MultipliersFoundations and Trends in Machine Learning, 2011
- Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling2009
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Context-dependent connectionist probability estimation in a hybrid hidden Markov model-neural net speech recognition systemComputer Speech & Language, 1994
- Learning representations by back-propagating errorsNature, 1986