Non-Invasive, Memory Access-Triggered Near-Data Processing for DNN Training Acceleration on GPUs
Open Access
- 23 September 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 142651-142667
Abstract
To address the off-chip bandwidth (BW) and memory capacity challenges of GPUs in DNN training, we propose a memory access-triggered near-data processing mat NDP architecture that offloads memory- and communication-bound operations. With mat NDP, normal memory accesses also serve as implicit NDP requests to enable NDP in a non-invasive manner without modifying core-side ISA/microarchitecture/SW, for practical NDP. In addition, mat NDP enables on-the-fly NDP where the data already supplied in normal memory requests for compute-bound operations are also simultaneously used for NDP; thus, matNDP can overlap even dependent kernels while also reducing memory traffic. Moreover, with the overlap, memory bandwidth (BW) underutilized by GPU cores can be used by NDP units to improve performance even under the same total memory BW. The matNDP units can be deployed to heterogeneous memory devices in a system. First, we deploy them near GPU’s memory controllers. Secondly, our NDP units are deployed in memory expanders connected to multiple GPUs to create an NDP-enabled memory eXpander Network (NDPXNet) . It can entirely offload gradient reduction and optimizer in data-parallel training, achieving additional speedups while eliminating redundancy in memory usage and optimizer execution. As a result, we 1) enable NDP without core HW/SW changes, 2) overlap the execution of dependent layers, and 3) offload both memory- and communicationbound operations from GPUs in DNN training. Through our deep learning compiler support, NDP kernels can be generated automatically without any model code modification. Consequently, mat NDP can improve training throughput by up to 2.73× and reduce energy by up to 41.4%.Keywords
Funding Information
- Institute for Information Communication Technology Planning and Evaluation (IITP_2024-RS-2024-00437866, RS-2019-II191906, RS-2021-II210310, RS-2021-II210871)
- National Research Foundation of Korea (RS-2023-00277080, RS-2024-00415602)
This publication has 93 references indexed in Scilit:
- NERO: A Near High-Bandwidth Memory Stencil Accelerator for Weather Prediction Modeling2020
- McDRAM v2: In-Dynamic Random Access Memory Systolic Array Accelerator to Address the Large Model Problem in Deep Neural Networks on the EdgeIEEE Access, 2020
- MnasNet: Platform-Aware Neural Architecture Search for Mobile2019
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Squeeze-and-Excitation Networks2018
- DrAcc: a DRAM based Accelerator for Accurate CNN Inference2018
- Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for Efficient Data Reduction2016
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015