Modular Spiking Neural Circuits for Mapping Long Short-Term Memory on a Neurosynaptic Processor
- 13 July 2018
- journal article
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
- Vol. 8 (4) , 782-795
Abstract
Due to the distributed and asynchronous nature of neural computation through low-energy spikes, brain-inspired hardware systems offer high energy efficiency and massive parallelism. One such platform is the IBM TrueNorth Neurosynaptic System. Recently, TrueNorth compatible representation learning algorithms have emerged, achieving close to state-of-the-art performance in various datasets. However, its application in temporal sequence processing models such as recurrent neural networks (RNNs) is still only at the proof of concept level. There is an inherent difficulty in capturing temporal dynamics of an RNN using spiking neurons which is only exasperated by the hardware constraints in connectivity and synaptic weight resolution. This work presents a design flow that overcomes these difficulties and maps a special case of recurrent networks called Long Short-Term Memory (LSTM) onto a spike-based platform. The framework utilizes various approximation techniques; activation discretization, weight quantization, scaling and rounding; spiking neural circuits that implement the complex gating mechanisms, and a store-and-release technique to enable neuron synchronization and faithful storage. While the presented techniques can be applied to map LSTM to any Spiking Neural Network (SNN) simulator/emulator, here we choose the TrueNorth chip as the target platform by adhering to its hardware constraints. Three LSTM applications, parity check, Extended Reber Grammar and Question classification, are evaluated. The tradeoffs among accuracy, performance and energy tradeoffs achieved on TrueNorth are demonstrated. This is compared to the performance on an SNN platform without hardware constraints, which represents the upper bound of the achievable accuracy.Keywords
Funding Information
- National Science Foundation (CNS-1739748)
This publication has 20 references indexed in Scilit:
- A spike-based long short-term memory on a neurosynaptic processor2017
- Balanced Quantization: An Effective and Efficient Approach to Quantized Neural NetworksJournal of Computer Science and Technology, 2017
- Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware2016
- Convolutional networks for fast, energy-efficient neuromorphic computingProceedings of the National Academy of Sciences, 2016
- Sequence to Sequence -- Video to Text2015
- On Training Efficiency and Computational Costs of a Feed Forward Neural Network: A ReviewComputational Intelligence and Neuroscience, 2015
- Show and tell: A neural image caption generator2015
- A million spiking-neuron integrated circuit with a scalable communication network and interfaceScience, 2014
- Neurogrid: A Mixed-Analog-Digital Multichip System for Large-Scale Neural SimulationsProceedings of the IEEE, 2014
- Learning long-term dependencies with gradient descent is difficultIEEE Transactions on Neural Networks, 1994