Conversion of artificial recurrent neural networks to spiking neural networks for low-power neuromorphic hardware
Authors:
- 10 November 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
In recent years the field of neuromorphic low-power systems gained significant momentum, spurring brain-inspired hardware systems which operate on principles that are fundamentally different from standard digital computers and thereby consume orders of magnitude less power. However, their wider use is still hindered by the lack of algorithms that can harness the strengths of such architectures. While neuromorphic adaptations of representation learning algorithms are now emerging, the efficient processing of temporal sequences or variable length-inputs remains difficult, partly due to challenges in representing and configuring the dynamics of spiking neural networks. Recurrent neural networks (RNN) are widely used in machine learning to solve a variety of sequence learning tasks. In this work we present a train-and-constrain methodology that enables the mapping of machine learned (Elman) RNNs on a substrate of spiking neurons, while being compatible with the capabilities of current and near-future neuromorphic systems. This “train-and-constrain” method consists of first training RNNs using backpropagation through time, then discretizing the weights and finally converting them to spiking RNNs by matching the responses of artificial neurons with those of the spiking neurons. We demonstrate our approach by mapping a natural language processing task (question classification), where we demonstrate the entire mapping process of the recurrent layer of the network on IBM's Neurosynaptic System TrueNorth, a spike-based digital neuromorphic hardware architecture. TrueNorth imposes specific constraints on connectivity, neural and synaptic parameters. To satisfy these constraints, it was necessary to discretize the synaptic weights to 16 levels, discretize the neural activities to 16 levels, and to limit fan-in to 64 inputs. Surprisingly, we find that short synaptic delays are sufficient to implement the dynamic (temporal) aspect of the RNN in the question classification task. Furthermore we observed that the discretization of the neural activities is beneficial to our train-and-constrain approach. The hardware-constrained model achieved 74% accuracy in question classification while using less than 0.025% of the cores on one TrueNorth chip, resulting in an estimated power consumption of ≈ 17µW.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- Delay learning architectures for memory and classificationNeurocomputing, 2014
- STDP Installs in Winner-Take-All Circuits an Online Approximation to Hidden Markov Model LearningPLOS Computational Biology, 2014
- Stochastic Computations in Cortical Microcircuit ModelsPLOS Computational Biology, 2013
- Deep, Big, Simple Neural Nets for Handwritten Digit RecognitionNeural Computation, 2010
- A VLSI Array of Low-Power Spiking Neurons and Bistable Synapses With Spike-Timing Dependent PlasticityIEEE Transactions on Neural Networks, 2006
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Long Short-Term MemoryNeural Computation, 1997
- Finding structure in timeCognitive Science, 1990
- Backpropagation through time: what it does and how to do itProceedings of the IEEE, 1990
- Excitatory and Inhibitory Interactions in Localized Populations of Model NeuronsBiophysical Journal, 1972