Channel and filter parallelism for large-scale CNN training
Authors:
Open Access
- 17 November 2019
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Abstract
Accelerating large-scale CNN training is needed to keep training times reasonable as datasets grow larger and models become more complex. Existing frameworks primarily scale using data-parallelism, but this is limited by the mini-batch size, which cannot grow arbitrarily. We introduce three algorithms that partition channel or filter data to exploit parallelism beyond the sample dimension. Further, they partition the parameters of convolutional layers, replacing global all reduces with segmented allreduces---smaller, concurrent allreduces among disjoint processor sets. These algorithms enable strong scaling, reduced communication overhead, and reduced memory pressure, enabling training of very wide CNNs. We demonstrate improved strong and weak scaling, including up to 4.1x reductions in training time for residual networks and 4x reductions in allreduce overhead. We also show that wider models provide improved accuracy on ImageNet. We study the current limitations of our algorithms and provide a direction for future optimizations of large-scale deep learning frameworks.Keywords
Funding Information
- LDRD
This publication has 28 references indexed in Scilit:
- Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability2016
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Fast Algorithms for Convolutional Neural Networks2016
- Parallel Matrix Multiplication: A Systematic JourneySIAM Journal on Scientific Computing, 2016
- LBANN2015
- Fully convolutional networks for semantic segmentation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- A Linear Algebra Framework for Static High Performance Fortran Code DistributionScientific Programming, 1995