RELL-STC: An Efficient SpMM Extension of Sparse Tensor Core with Re-structured ELLPACK and Gustavson Dataflow
Open Access
- 29 May 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Measurement and Analysis of Computing Systems
- Vol. 10 (2) , 1-22
Abstract
Sparsity is a crucial optimization technique for deep neural networks, significantly reducing storage and bandwidth requirements while enhancing computational efficiency. In recent years, driven by the rapid advancement of deep neural networks, accelerators targeting sparse computation have emerged as a significant area of research. NVIDIA introduced 2:4 fine-grained structured sparsity starting with the Ampere architecture, which doubles the peak throughput of Tensor Cores by retaining only two out of every four elements. However, the constraints imposed by structured sparsity on parameters may compromise model accuracy. In contrast, unstructured sparsity enables maximum sparsity while maintaining model accuracy. Nevertheless, the irregular distribution of non-zero elements (NNZs) in unstructured sparse matrices introduces complex memory access patterns and load imbalances, preventing the direct utilization of GPU Tensor Cores. Consequently, computation typically relies on CUDA cores via sparse libraries such as cuSPARSE, leaving the powerful computational capabilities of Tensor Cores underutilized. To address this limitation, we extend the Tensor Core architecture inspired by Gustavson's dataflow and propose a re-structured row offline compression format tailored for the inference stage. This format integrates seamlessly with the extended architecture to achieve highly efficient unstructured sparse-dense matrix multiplication. Experimental results demonstrate that with negligible area and power overheads, our method achieves an average speedup of 3.54x compared to cuSPARSE and 1.2x compared to state-of-the-art Tensor Core extensions.Keywords
Funding Information
- National High-Level Talent Special Support Programs (10,000 Talents Program) - Young Talents (No.ZD0202082503)
This publication has 24 references indexed in Scilit:
- Dual-side Sparse Tensor Core2021
- SPAGHETTI: Streaming Accelerators for Highly Sparse GEMM on FPGAs2021
- MatRaptor: A Sparse-Sparse Matrix Multiplication Accelerator Based on Row-Wise Product2020
- Accel-Sim: An Extensible Simulation Framework for Validated GPU Modeling2020
- SpArch: Efficient Architecture for Sparse Matrix Multiplication2020
- SIGMA: A Sparse and Irregular GEMM Accelerator with Flexible Interconnects for DNN Training2020
- SMT-SA: Simultaneous Multithreading in Systolic ArraysIEEE Computer Architecture Letters, 2019
- OuterSPACE: An Outer Product Based Sparse Matrix Multiplication Accelerator2018
- Analyzing CUDA workloads using a detailed GPU simulator2009
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998