BRP-SpMM: Block-Row Partition Based Sparse Matrix Multiplication with Tensor and CUDA Cores
Authors:
- 3 June 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 901-912
Abstract
Sparse-Dense Matrix Multiplication (SpMM) is a fundamental computational operation in various domains, and leveraging Tensor cores or CUDA cores on GPU to accelerate SpMM has become common practice. While Tensor cores offer notable advantages in dense matrix multiplication, their efficiency significantly decreases when handling sparse matrices. Besides, the computational power of CUDA cores should not be overlooked. Nevertheless, the differences in supported data formats present challenges in effectively leveraging both types of cores to accelerate SpMM. To this end, we propose BRP-SpMM, a block and row partition approach designed for efficient SpMM on GPU. BRP-Sp partitions the sparse matrix into two parts: TC Block and Residual Row part, which are computed on Tensor cores and CUDA cores separately. Meanwhile, a customized storage format is proposed to manage these two distinct parts. Our two GPU kernels incorporate several advanced techniques including load balance, register remapping, and 1-D tiling. BRP-SpMM can achieve higher memory access efficiency and more rational computing resource utilization of GPU. Extensive experiments on modern NVIDIA A800 GPU show that BRPSpMM outperforms SOTA libraries by up to $2.9 \times$ (on average $2.1 \times)$. Furthermore, BRP-SpMM accelerates end-to-end GNN training by up to $1.9 \times$ compared to popular frameworks.
Keywords
Funding Information
- National Key Research and Development Program of China (2022YFB4501400)
- National Natural Science Foundation of China (62372199)
This publication has 18 references indexed in Scilit:
- Exploiting Tensor Cores in Sparse Matrix-Multivector Multiplication via Block-Sparsity-Aware Clustering2024
- GLARE: Accelerating Sparse DNN Inference Kernels with Global Memory Access Reduction2023
- Adaptive Sparse Deep Neural Network Inference on Resource-Constrained Cost-Efficient GPUs2023
- Efficient Quantized Sparse Matrix Operations on Tensor Cores2022
- Blocking Sparse Matrices to Leverage Dense-Specific Multiplication2022
- Accelerating Sparse Deep Neural Network Inference Using GPU Tensor Cores2022
- Fast Sparse Deep Neural Network Inference with Flexible SpMM Optimization Space Exploration2021
- Sparse GPU Kernels for Deep Learning2020
- A matrix approach to tomographic reconstruction and its implementation on GPUsJournal of Structural Biology, 2010
- Matrix Krylov subspace methods for linear systems with multiple right-hand sidesNumerical Algorithms, 2005