Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks
Authors:
Open Access
- 14 July 2022
- journal article
- research article
- Published by IOP Publishing in Neuromorphic Computing and Engineering
- Vol. 2 (3) , 034004
Abstract
In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the potential to reduce the computational cost of neural networks by two orders of magnitude. Despite this potential, today’s neural networks deliver only modest performance benefits using just weight sparsity, because traditional computing hardware cannot efficiently process sparse networks. In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100× improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2. Our results with Complementary Sparsity suggest that weight plus activation sparsity can be a potent combination for efficiently scaling future AI models.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- Sparse Coding via Thresholding and Local Competition in Neural CircuitsNeural Computation, 2008
- Small-World Brain NetworksThe Neuroscientist, 2006
- Efficient auditory codingNature, 2006
- Excitatory cortical neurons form fine-scale functional networksNature, 2005
- Sparse coding of sensory inputsCurrent Opinion in Neurobiology, 2004
- Pyramidal cell communication within local networks in layer 2/3 of rat neocortexThe Journal of Physiology, 2003
- The Cost of Cortical ComputationCurrent Biology, 2003
- Coding of Natural Scenes in Primary Visual CortexNeuron, 2003
- An Energy Budget for Signaling in the Grey Matter of the BrainJournal of Cerebral Blood Flow & Metabolism, 2001
- Emergence of simple-cell receptive field properties by learning a sparse code for natural imagesNature, 1996