DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
Preprint
Abstract
Synchronous strategies with data parallelism, such as the Synchronous StochasticGradient Descent (S-SGD) and the model averaging methods, are widely utilizedin distributed training of Deep Neural Networks (DNNs), largely owing to itseasy implementation yet promising performance. Particularly, each worker ofthe cluster hosts a copy of the DNN and an evenly divided share of the datasetwith the fixed mini-batch size, to keep the training of DNNs convergence. In thestrategies, the workers with different computational capability, need to wait foreach other because of the synchronization and delays in network transmission,which will inevitably result in the high-performance workers wasting computation.Consequently, the utilization of the cluster is relatively low. To alleviate thisissue, we propose the Dynamic Batch Size (DBS) strategy for the distributedtraining of DNNs. Specifically, the performance of each worker is evaluatedfirst based on the fact in the previous epoch, and then the batch size and datasetpartition are dynamically adjusted in consideration of the current performanceof the worker, thereby improving the utilization of the cluster. To verify theeffectiveness of the proposed strategy, extensive experiments have been conducted,and the experimental results indicate that the proposed strategy can fully utilizethe performance of the cluster, reduce the training time, and have good robustnesswith disturbance by irrelevant tasks. Furthermore, rigorous theoretical analysis hasalso been provided to prove the convergence of the proposed strategy.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Stochastic quasi-gradient methods: variance reduction via Jacobian sketchingMathematical Programming, 2020
- Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep LearningProceedings of the AAAI Conference on Artificial Intelligence, 2019
- A Distributed Synchronous SGD Algorithm with Global Top-k Sparsification for Low Bandwidth Networks2019
- An Adaptive Synchronous Parallel Strategy for Distributed Machine LearningIEEE Access, 2018
- Kylix: A Sparse Allreduce for Commodity Clusters2014
- Parallel stochastic gradient algorithms for large-scale matrix completionMathematical Programming Computation, 2013