Distributed Learning of Deep Neural Networks using Independent Subnet Training
Authors:
- Binhang YuanBYBinhang Yuan
- Anastasios KyrillidisAKAnastasios Kyrillidis
- Christopher M. JermaineCJChristopher M. Jermaine
Preprint
Abstract
We alleviate costly communication/computation overhead in classical distributed learning by introducing independent subnet training: a novel, simple, jointly model-parallel and data-parallel approach to distributed neural network training.Our main idea is that, per iteration, the model's neurons can be randomly divided into smaller surrogate models without replacement---dubbed as subnetworks or subnets---and each subwork is sent for training only to a single worker. This way, our algorithm broadcasts the whole model parameters only once into the distributed network per synchronization cycle. This not only reduces the overall communication overhead, but also the computation workload: each worker only receives the weights associated with the subwork it has been assigned to. Further, subwork generation and training reduces synchronization frequency: since workers train disjoint portions of the network as if they were independent models, training continues for longer periods of time before synchronization, similar to local SGD approaches. We test our approach on speech recognition and product recommendation applications, as well as image classification tasks. Subnet training: i) leads to accelerated training, as compared to state of the art distributed models, and ii) often results into boosting the testing accuracy, as it implicitly leverages dropout regularization during training.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Inefficiency of K-FAC for Large Batch Size TrainingProceedings of the AAAI Conference on Artificial Intelligence, 2020
- An investigation of Newton-Sketch and subsampled Newton methodsOptimization Methods and Software, 2020
- Large-batch training for LSTM and beyond2019
- Newton-type methods for non-convex optimization under inexact Hessian informationMathematical Programming, 2019
- GPU Accelerated Sub-Sampled Newton's Method for Convex Classification Problems2019
- Optimization Methods for Large-Scale Machine LearningSIAM Review, 2018
- Sparse Communication for Distributed Gradient Descent2017
- Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix MultiplicationSIAM Journal on Computing, 2006
- A Scale for the Measurement of the Psychological Magnitude PitchThe Journal of the Acoustical Society of America, 1937