Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks
Authors:
- Minsoo RhuMRMinsoo Rhu
- Mike O'COnnorMOMike O'COnnor
- Niladrish ChatterjeeNCNiladrish Chatterjee
- Jeff PoolJPJeff Pool
- Youngeun KwonYKYoungeun Kwon
- Stephen W. KecklerSKStephen W. Keckler
- 1 February 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 78-91
Abstract
Popular deep learning frameworks require users to fine-tune their memory usage so that the training data of a deep neural network (DNN) fits within the GPU physical memory. Prior work tries to address this restriction by virtualizing the memory usage of DNNs, enabling both CPU and GPU memory to be utilized for memory allocations. Despite its merits, virtualizing memory can incur significant performance overheads when the time needed to copy data back and forth from CPU memory is higher than the latency to perform DNN computations. We introduce a high-performance virtualization strategy based on a "compressing DMA engine" (cDMA) that drastically reduces the size of the data structures that are targeted for CPU-side allocations. The cDMA engine offers an average 2.6x (maximum 13.8x) compression ratio by exploiting the sparsity inherent in offloaded data, improving the performance of virtualized DNNs by an average 53% (maximum 79%) when evaluated on an NVIDIA Titan Xp.Keywords
All Related Versions
This publication has 36 references indexed in Scilit:
- DjiNN and Tonic2015
- Deeply learned face representations are sparse, selective, and robust2015
- DaDianNao: A Machine-Learning Supercomputer2014
- Architectural support for address translation on GPUs2014
- DianNaoACM SIGPLAN Notices, 2014
- Supporting x86-64 address translation for 100s of GPU lanes2014
- Gzip on a chip2014
- Lossless and lossy memory I/O link compression for improving performance of GPGPU workloads2012
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Long Short-Term MemoryNeural Computation, 1997