SAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs
Authors:
- 1 October 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 796-810
Abstract
General Matrix Multiplication (GEMM) is the key operation in Deep Neural Networks (DNNs). While dense GEMM uses SIMD CPUs efficiently, sparse GEMM is much less efficient, especially at the modest levels of unstructured sparsity common in DNN inference/training. Thus, most DNNs use dense GEMM.In this paper, we propose SAVE, a novel vector engine for CPUs that efficiently skips ineffectual computation due to sparsity in dense DNN implementations. SAVE’s hardware extensions to the vector pipeline are transparent to software. SAVE accelerates FP32 and mixed-precision kernels with unstructured sparsity from both weights and activations. Further, SAVE is not DNN-specific and can potentially speed-up any vector workload with sparsity. To evaluate SAVE, we use simulations of a 28-core machine and run VGG16, ResNet-50, and GNMT, with and without pruning. With realistic sparsity, SAVE accelerates inference by 1.37x-1.68x and end-to-end training by 1.28x-1.64x.Keywords
This publication has 30 references indexed in Scilit:
- CGBVS‐DNN: Prediction of Compound‐protein Interactions Based on Deep LearningMolecular Informatics, 2016
- CnvlutinACM SIGARCH Computer Architecture News, 2016
- EyerissACM SIGARCH Computer Architecture News, 2016
- EIEACM SIGARCH Computer Architecture News, 2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Going deeper with convolutions2015
- Maximizing SIMD resource utilization in GPGPUs with SIMD lane permutation2013
- Sniper2011
- Learning to Forget: Continual Prediction with LSTMNeural Computation, 2000
- Vector instruction set support for conditional operationsACM SIGARCH Computer Architecture News, 2000