Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design
- 20 September 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 43 (2) , 506-519
Abstract
Sparse training is one of the promising techniques to reduce the computational cost of deep neural networks (DNNs) while retaining high accuracy. In particular, N:M fine-grained structured sparsity, where only ${N}$ out of consecutive ${M}$ elements can be nonzero, has attracted attention due to its hardware-friendly pattern and capability of achieving a high sparse ratio. However, the potential to accelerate N:M sparse DNN training has not been fully exploited, and there is a lack of efficient hardware supporting N:M sparse training. To tackle these challenges, this article presents a computation-efficient training scheme for N:M sparse DNNs using algorithm, architecture, and dataflow co-design. At the algorithm level, a bidirectional weight pruning method, dubbed BDWP, is proposed to leverage the N:M sparsity of weights during both forward and backward passes of DNN training, which can significantly reduce the computational cost while maintaining model accuracy. At the architecture level, a sparse accelerator for DNN training, namely, SAT, is developed to neatly support both the regular dense operations and the computation-efficient N:M sparse operations. At the dataflow level, multiple optimization methods ranging from interleave mapping, pregeneration of N:M sparse weights, and offline scheduling, are proposed to boost the computational efficiency of SAT. Finally, the effectiveness of our training scheme is evaluated on a Xilinx VCU1525 FPGA card using various DNN models (ResNet9, ViT, VGG19, ResNet18, and ResNet50) and datasets (CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet). Experimental results show the SAT accelerator with the BDWP sparse training method under 2:8 sparse ratio achieves an average speedup of $1.75\times $ over that with the dense training, accompanied by a negligible accuracy loss of 0.56% on average. Furthermore, our proposed training scheme significantly improves the training throughput by $2.97\times $ – $25.22\times $ and the energy efficiency by $1.36\times $ – $3.58\times $ over prior FPGA-based accelerators.
Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62174084)
- National Key Research and Development Program of China (2022YFB4400604)
This publication has 48 references indexed in Scilit:
- Towards efficient deep neural network training by FPGA-based batch-level parallelismJournal of Semiconductors, 2020
- Pre-defined Sparsity for Low-Complexity Convolutional Neural NetworksIEEE Transactions on Computers, 2020
- Low-bit Quantization of Neural Networks for Efficient Inference2019
- FPGA-Based Training Accelerator Utilizing Sparseness of Convolutional Neural Network2019
- Automatic Compiler Based FPGA Accelerator for CNN Training2019
- Maestro: A Memory-on-Logic Architecture for Coordinated Parallel Use of Many Systolic Arrays2019
- A Low-latency Sparse-Winograd Accelerator for Convolutional Neural Networks2019
- Approximate Random Dropout for DNN training acceleration in GPGPU2019
- In-Datacenter Performance Analysis of a Tensor Processing Unit2017
- Deep Residual Learning for Image Recognition2016