TSUNAMI: Triple Sparsity-Aware Ultra Energy-Efficient Neural Network Training Accelerator With Multi-Modal Iterative Pruning
- 4 January 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems I: Regular Papers
- Vol. 69 (4) , 1494-1506
Abstract
This article proposes the TSUNAMI, which supports an energy-efficient deep-neural-network training. The TSUNAMI supports multi-modal iterative pruning to generate zeros in activation and weight. Tile-based dynamic activation pruning unit and weight memory shared pruning unit eliminate additional memory access. Coarse-zero skipping controller skips multiple unnecessary multiply-and-accumulation (MAC) operations at once, and fine-zero skipping controller skips randomly located unnecessary MAC operations. Weight sparsity balancer solves a utilization degradation caused by weight sparsity imbalance, and the workload of each convolution core is allocated by a random channel allocator. The TSUNAMI achieves an energy efficiency of 3.42 TFLOPS/W at 0.78V and 50MHz with floating-point 8-bit activation and weight. Also, it achieves an energy efficiency of 405.96 TFLOPS/W at 90% sparsity condition.Keywords
Funding Information
- Institute of Information and Communications Technology Planning and Evaluation (IITP) Grant by the Korean Government through MSIT (2021-0-00871)
This publication has 23 references indexed in Scilit:
- 7.7 LNPU: A 25.3TFLOPS/W Sparse Deep-Neural-Network Learning Processor with Fine-Grained Mixed Precision of FP8-FP162019
- A 2.1TFLOPS/W Mobile Deep RL Accelerator with Transposable PE Array and Experience Compression2019
- UNPU: A 50.6TOPS/W unified deep neural network accelerator with 1b-to-16b fully-variable weight bit-precision2018
- A 141.4 mW Low-Power Online Deep Neural Network Training Processor for Real-time Object Tracking in Mobile Devices2018
- A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of ThingsIEEE Transactions on Circuits and Systems I: Regular Papers, 2017
- 14.2 DNPU: An 8.1TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks2017
- Fused-layer CNN accelerators2016
- Describing Textures in the Wild2014
- Bag-of-visual-words and spatial extensions for land-use classification2010
- ImageNet: A large-scale hierarchical image database2009