moDNN: Memory Optimal Deep Neural Network Training on Graphics Processing Units
- 23 August 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 30 (3) , 646-661
Abstract
Graphics processing units (GPUs) have been widely adopted to accelerate the training of deep neural networks (DNNs). Although the computational performance of GPUs has been improving steadily, the memory size of modern GPUs is still quite limited, which restricts the sizes of DNNs that can be trained on GPUs, hence raises serious challenges. This paper introduces a framework, referred to as moDNN (memory optimal DNN training on GPUs), to optimize the memory usage in DNN training. moDNN supports automatic tuning of DNN training code to match any given memory budget (larger than the theoretical lower bound). By taking full advantage of overlapping computations and data transfers, we develop new heuristics to judiciously schedule data offloading and prefetching transfers, together with convolution algorithm selection, to optimize memory usage. We further devise a new sub-batch size selection method which also greatly reduces memory usage. moDNN can save memory usage up to 59X, compared with an ideal case which assumes that the GPU memory is sufficient to hold all data. When executing moDNN on a GPU with 12GB memory, the training time is increased by only 3%, which is much shorter than that incurred by the best known approach, vDNN. Furthermore, we propose an optimization strategy for moDNN on multiple GPUs again by utilizing the idea of overlapping data transfers and GPU computations. The results show that 3.7X speedup is attained on four GPUs.Keywords
Funding Information
- National Science Foundation (CCF-1217906, CNS-1629914, CCF-1617735, CCF-1640081)
- Nanoelectronics Research Corporation
- Semiconductor Research Corporation
- Extremely Energy Efficient Collective Electronics
- SRC-NRI Nanoelectronics Research Initiative (2698.004, 2698.005)
This publication has 12 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- Fast Algorithms for Convolutional Neural Networks2016
- Going deeper with convolutions2015
- Fully convolutional networks for semantic segmentation2015
- Caffe2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- External memory algorithms and data structuresACM Computing Surveys, 2001
- Conjugate gradient algorithm for efficient training of artificial neural networksIEE Proceedings G Circuits, Devices and Systems, 1992
- A comparison of next-fit, first-fit, and best-fitCommunications of the ACM, 1977