VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Preprint
Abstract
The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compute efficiency, and accelerated hardware is becoming available. However, exploiting it efficiently requires kernel implementations, pruning algorithms, and storage formats, to utilize hardware support of specialized sparse vector units. An example of those are the NVIDIA's Sparse Tensor Cores (SPTCs), which promise a 2x speedup. However, SPTCs only support the 2:4 format, limiting achievable sparsity ratios to 50%. We present the V:N:M format, which enables the execution of arbitrary N:M ratios on SPTCs. To efficiently exploit the resulting format, we propose Spatha, a high-performance sparse-library for DL routines. We show that Spatha achieves up to 37x speedup over cuBLAS. We also demonstrate a second-order pruning technique that enables sparsification to high sparsity ratios with V:N:M and little to no loss in accuracy in modern transformers.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Dynamic N:M Fine-Grained Structured Sparse Attention Mechanism2023
- Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric BehaviorsIEEE Transactions on Parallel and Distributed Systems, 2022
- Probing the Efficacy of Hardware-Aware Weight Pruning to Optimize the SpMM Routine on Ampere GPUs2022
- Efficient tensor core-based GPU kernels for structured sparsity under reduced precision2021
- Block Pruning For Faster Transformers2021
- BERT Busters: Outlier Dimensions that Disrupt Transformers2021
- Sparse GPU Kernels for Deep Learning2020
- Demystifying Tensor Cores to Optimize Half-Precision Matrix Multiply2020
- Structured Pruning of Large Language Models2020
- Improving performance of sparse matrix-vector multiplication1999