VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores
Open Access
- 11 November 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-14
Abstract
The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compute efficiency, and accelerated hardware is becoming available. However, exploiting it efficiently requires kernel implementations, pruning algorithms, and storage formats, to utilize hardware support of specialized sparse vector units. An example of those are the NVIDIA's Sparse Tensor Cores (SPTCs), which promise a 2× speedup. However, SPTCs only support the 2:4 format, limiting achievable sparsity ratios to 50%. We present the V:N:M format, which enables the execution of arbitrary N:M ratios on SPTCs. To efficiently exploit the resulting format, we propose Spatha, a high-performance sparse-library for DL routines. We show that Spatha achieves up to 37× speedup over cuBLAS. We also demonstrate a second-order pruning technique that enables sparsification to high sparsity ratios with V:N:M and little to no loss in accuracy in modern transformers.Keywords
All Related Versions
Funding Information
- Ministerio de Ciencia e Innovación (PID2019-104184RB-I00)
- Ministerio de Educación y Formación Profesional (FPU19/03974)
- Xunta de Galicia (ED431C 2021/30, ED431G 2019/01)
This publication has 13 references indexed in Scilit:
- Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric BehaviorsIEEE Transactions on Parallel and Distributed Systems, 2022
- Probing the Efficacy of Hardware-Aware Weight Pruning to Optimize the SpMM Routine on Ampere GPUs2022
- Efficient tensor core-based GPU kernels for structured sparsity under reduced precision2021
- Block Pruning For Faster Transformers2021
- BERT Busters: Outlier Dimensions that Disrupt Transformers2021
- Demystifying Tensor Cores to Optimize Half-Precision Matrix Multiply2020
- Structured Pruning of Large Language Models2020
- Energy and Policy Considerations for Deep Learning in NLP2019
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned2019
- Improving performance of sparse matrix-vector multiplication1999