VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUs
Preprint
Abstract
Deep Learning (DL) acceleration support in CPUs has recently gained a lot of traction, with several companies (Arm, Intel, IBM) announcing products with specialized matrix engines accessible via GEMM instructions. CPUs are pervasive and need to handle diverse requirements across DL workloads running in edge/HPC/cloud platforms. Therefore, as DL workloads embrace sparsity to reduce the computations and memory size of models, it is also imperative for CPUs to add support for sparsity to avoid under-utilization of the dense matrix engine and inefficient usage of the caches and registers. This work presents VEGETA, a set of ISA and microarchitecture extensions over dense matrix engines to support flexible structured sparsity for CPUs, enabling programmable support for diverse DL models with varying degrees of sparsity. Compared to the state-of-the-art (SOTA) dense matrix engine in CPUs, a VEGETA engine provides 1.09x, 2.20x, 3.74x, and 3.28x speed-ups when running 4:4 (dense), 2:4, 1:4, and unstructured (95%) sparse DNN layers.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- AWB-GCN: A Graph Convolutional Network Accelerator with Runtime Workload Rebalancing2020
- SAVE: Sparsity-Aware Vector Engine for Accelerating DNN Training and Inference on CPUs2020
- Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN InferenceIEEE Computer Architecture Letters, 2020
- ExTensor2019
- Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile DevicesIEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2019
- Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations2019
- SparCE: Sparsity Aware General-Purpose Core Extensions to Accelerate Deep Neural NetworksIEEE Transactions on Computers, 2018
- OuterSPACE: An Outer Product Based Sparse Matrix Multiplication Accelerator2018
- Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural NetworksIEEE Journal of Solid-State Circuits, 2016
- PinACM SIGPLAN Notices, 2005