Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices
Authors:
Preprint
Abstract
In a previous work we have detailed the requirements to obtain a maximal performance benefit by implementing fully connected deep neural networks (DNN) in form of arrays of resistive devices for deep learning. This concept of Resistive Processing Unit (RPU) devices we extend here towards convolutional neural networks (CNNs). We show how to map the convolutional layers to RPU arrays such that the parallelism of the hardware can be fully utilized in all three cycles of the backpropagation algorithm. We find that the noise and bound limitations imposed due to analog nature of the computations performed on the arrays effect the training accuracy of the CNNs. Noise and bound management techniques are presented that mitigate these problems without introducing any additional complexity in the analog circuits and can be addressed by the digital circuits. In addition, we discuss digitally programmable update management and device variability reduction techniques that can be used selectively for some of the layers in a CNN. We show that combination of all those techniques enables a successful application of the RPU concept for training CNNs. The techniques discussed here are more general and can be applied beyond CNN architectures and therefore enables applicability of RPU approach for large class of neural network architectures.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Neuromorphic computing using non-volatile memoryAdvances in Physics: X, 2016
- Li‐Ion Synaptic Transistor for Low Power Analog ComputingAdvanced Materials, 2016
- Repeatable, accurate, and high speed multi-level programming of memristor 1T1R arrays for power efficient analog computing applicationsNanotechnology, 2016
- Training and operation of an integrated neuromorphic network based on metal-oxide memristorsNature, 2015
- Parallel Programming of Resistive Cross-point Array for Synaptic PlasticityProcedia Computer Science, 2014
- Synaptic electronics: materials, devices and applicationsNanotechnology, 2013
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- A generic systolic array building block for neural networks with on-chip learningIEEE Transactions on Neural Networks, 1993
- A 336-neuron, 28 K-synapse, self-learning neural network chip with branch-neuron-unit architectureIEEE Journal of Solid-State Circuits, 1991
- Die LernmatrixBiological Cybernetics, 1961