Efficient Tensor Offloading for Large Deep-Learning Model Training based on Compute Express Link
Authors:
- 17 November 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-18
Abstract
The deep learning models (DL) are becoming bigger, easily beyond the memory capacity of a single accelerator. The recent progress in large DL training utilizes CPU memory as an extension of accelerator memory and offloads tensors to CPU memory to save accelerator memory. This solution transfers tensors between the two memories, creating a major performance bottleneck. We identify two problems during tensor transfers: (1) the coarse-grained tensor transfer creating difficulty in hiding transfer overhead, and (2) the redundant transfer that unnecessarily migrates value-unchanged bytes from CPU to accelerator. We introduce a cache coherence interconnect based on Compute Express Link (CXL) to build a cache coherence domain between CPU memory and accelerator memory. By slightly extending CXL to support an update cache-coherence protocol and avoiding unnecessary data transfers, we reduce training time by $33.7 \%$ (up to $55.4 \%$) without changing model convergence and accuracy, compared with the state-of-the-art work in DeepSpeed [62].
Keywords
Funding Information
- National Science Foundation
This publication has 55 references indexed in Scilit:
- Enabling Efficient Large-Scale Deep Learning Training with Cache Coherent Disaggregated Memory Systems2022
- RaPiD: AI Accelerator for Ultra-low Precision Training and Inference2021
- Stream Floating: Enabling Proactive and Decentralized Cache Optimizations2021
- Sentinel: Efficient Tensor Migration and Allocation on Heterogeneous Memory Systems for Deep Learning2021
- Accel-Sim: An Extensible Simulation Framework for Validated GPU Modeling2020
- Basic Enhancement Strategies When Using Bayesian Optimization for Hyperparameter Tuning of Deep Neural NetworksIEEE Access, 2020
- Dynamic Memory Management for GPU-Based Training of Deep Neural Networks2019
- Performance Modeling for Optimal Data Placement on GPU with Heterogeneous Memory Systems2017
- Algorithm-Directed Crash Consistence in Non-volatile Memory for HPC2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences of the United States of America, 2017