Abstract
Sparsity is crucial for deep neural networks, reducing storage and bandwidth while improving computational efficiency. With the rapid development of deep neural networks, sparse computation accelerators have become a key research focus. NVIDIA introduced 2:4 fine-grained structured sparsity in the Ampere architecture, doubling Tensor Core peak throughput by retaining two out of four elements, but this may compromise model accuracy. In contrast, unstructured sparsity achieves maximum sparsity without accuracy loss. However, the irregular distribution of non-zero elements (NNZs) in unstructured sparse matrices causes complex memory access and load imbalance, preventing direct use of GPU Tensor Cores. Thus, computation relies on CUDA cores via sparse libraries like cuSPARSE, underutilizing Tensor Cores' capabilities. To address this, we extend the Tensor Core architecture based on Gustavson's dataflow and propose a re-structured row offline compression format for inference. This format integrates with the extended architecture to enable efficient unstructured sparse-dense matrix multiplication. Experiments show negligible area and power overheads, with an average speedup of 3.54x over cuSPARSE and 1.2x over state-of-the-art Tensor Core extensions.