ATP: Achieving Throughput Peak for DNN Training via Smart GPU Memory Management
Open Access
- 20 March 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Architecture and Code Optimization
- Vol. 22 (1) , 1-27
Abstract
Due to the limited GPU memory, the performance of large DNNs training is constrained by the unscalable batch size. Existing studies partially address the issue of GPU memory limit through tensor recomputation and swapping, but overlook the exploration of optimal performance. In response, we propose ATP, a recomputation and swapping based GPU memory management framework that aims to maximize training performance by breaking GPU memory constraints. ATP utilizes a throughput model and we propose to evaluate the theoretical peak performance achievable by DNN training on GPU, and provide the optimum memory size required for recomputation and swapping. We optimize the mechanisms for GPU memory pool and CUDA stream control, employ an optimization method to search for specific tensors requiring recomputation and swapping, thereby bringing the actual DNN training performance on ATP closer to theoretical values. Evaluations with different types of large DNN models indicate that ATP achieve throughput improvements ranging from 1.14∼ 1.49×, while support model training exceeding the GPU memory limit by up to 9.2×.Keywords
This publication has 16 references indexed in Scilit:
- Multi-Tier GPU Virtualization for Deep Learning in Cloud-Edge SystemsIEEE Transactions on Parallel and Distributed Systems, 2023
- An Intelligent Framework for Oversubscription Management in CPU-GPU Unified MemoryJournal of Grid Computing, 2023
- Deep learning based data prefetching in CPU-GPU unified virtual memoryJournal of Parallel and Distributed Computing, 2022
- TENSILE: A Tensor Granularity Dynamic GPU Memory Scheduling Method Toward Multiple Dynamic Workloads SystemIEEE Transactions on Knowledge and Data Engineering, 2022
- A review on deep learning in medical image analysisInternational Journal of Multimedia Information Retrieval, 2021
- Performance evaluation of convolutional neural network on Tianhe-3 prototypeThe Journal of Supercomputing, 2021
- Deep Learning for Generic Object Detection: A SurveyInternational Journal of Computer Vision, 2019
- moDNN: Memory Optimal Deep Neural Network Training on Graphics Processing UnitsIEEE Transactions on Parallel and Distributed Systems, 2018
- ImageNet: A large-scale hierarchical image database2009
- Learning representations by back-propagating errorsNature, 1986