TSPLIT: Fine-grained GPU Memory Management for Efficient DNN Training via Tensor Splitting
Authors:
- 1 May 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2615-2628
Abstract
Since Deep Neural Networks (DNNs) are deeper and larger, performing DNNs training on existing accelerators (e.g., GPUs) is challenging due to their limited device memory capacity. Existing memory management systems reduce the mem-ory footprint via tensor offloading and recomputing. However, this coarse-grained, one-tensor-at-a-time memory management often incurs high peak GPU memory usage and cannot fully utilize available hardware resources (e.g., PCIe). In this paper, we propose TSPLIT, a fine-grained DNN memory management system that breaks apart memory bottlenecks while maintaining the efficiency of DNNs training. TSPLIT achieves this by proposing a model-guided approach to holistically exploit the tensor-split and its joint optimization with out-of-core execution methods (via offload and recompute). We further provide an efficient implementation of TSPLIT with proposed splittable tensor abstraction, profiling-based planner, and optimized DNN runtime. Evaluations on 6 DNN models show that compared to vDNN and SuperNeurons, TSPLIT can achieve maximum model scale up to 10.5× and 3.1 x and throughput improved up to 4.7× and 2.7 × under the same memory over-subscription, respectively.Keywords
This publication has 21 references indexed in Scilit:
- Reveal training performance mystery between TensorFlow and PyTorch in the single GPU environmentScience China Information Sciences, 2021
- Special focus on deep learning for computer visionScience China Information Sciences, 2020
- Split-CNN2019
- Superneurons2018
- Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks2018
- vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design2016
- Deep Residual Learning for Image Recognition2016
- Architectural support for address translation on GPUs2014
- Lookup Tables: Fine-Grained Partitioning for Distributed Databases2012
- Worst Case Analysis of Two Scheduling AlgorithmsSIAM Journal on Computing, 1977