The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
- 30 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. PP (10459219) , 1
Abstract
We present scalable hybrid-parallel algorithms for training large-scale 3D convolutional neural networks. Deep learning-based emerging scientific workflows often require model training with large, high-dimensional samples, which can make training much more costly and even infeasible due to excessive memory usage. We solve these challenges by extensively applying hybrid parallelism throughout the end-to-end training pipeline, including both computations and I/O. Our hybrid-parallel algorithm extends the standard data parallelism with spatial parallelism, which partitions a single sample in the spatial domain, realizing strong scaling beyond the mini-batch dimension with a larger aggregated memory capacity. We evaluate our proposed training algorithms with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive performance studies show that good weak and strong scaling can be achieved for both networks using up 2K GPUs. More importantly, we enable training of CosmoFlow with much larger samples than previously possible, realizing an order-of-magnitude improvement in prediction accuracy.Keywords
All Related Versions
Funding Information
- Japan Society for the Promotion of Science (Research Fellow of Japan Society for the Promotion)
- U.S. Department of Energy (U.S. DOE ECP ExaLearn Project)
This publication has 29 references indexed in Scilit:
- ooc_cuDNN: Accommodating convolutional neural networks over GPU memory capacity2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Predicting statistics of asynchronous SGD parameters for a large-scale distributed deep learning system on GPU supercomputers2016
- vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design2016
- V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- LBANN2015
- New Approaches in Turbulence and Transition Modeling Using Data-driven Techniques2015
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005