Accelerating Deep Learning with Memcomputing
Preprint
Abstract
Restricted Boltzmann machines (RBMs) and their extensions, called 'deep-belief networks', are powerful neural networks that have found applications in the fields of machine learning and artificial intelligence. The standard way to training these models resorts to an iterative unsupervised procedure based on Gibbs sampling, called 'contrastive divergence' (CD), and additional supervised tuning via back-propagation. However, this procedure has been shown not to follow any gradient and can lead to suboptimal solutions. In this paper, we show an efficient alternative to CD by means of simulations of digital memcomputing machines (DMMs). We test our approach on pattern recognition using a modified version of the MNIST data set. DMMs sample effectively the vast phase space given by the model distribution of the RBM, and provide a very good approximation close to the optimum. This efficient search significantly reduces the number of pretraining iterations necessary to achieve a given level of accuracy, as well as a total performance gain over CD. In fact, the acceleration of pretraining achieved by simulating DMMs is comparable to, in number of iterations, the recently reported hardware application of the quantum annealing method on the same network and data set. Notably, however, DMMs perform far better than the reported quantum annealing results in terms of quality of the training. We also compare our method to advances in supervised training, like batch-normalization and rectifiers, that work to reduce the advantage of pretraining. We find that the memcomputing method still maintains a quality advantage ($>1\%$ in accuracy, and a $20\%$ reduction in error rate) over these approaches. Furthermore, our method is agnostic about the connectivity of the network. Therefore, it can be extended to train full Boltzmann machines, and even deep networks at once.
Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Perspective: Memcomputing: Leveraging memory and physics to compute efficientlyJournal of Applied Physics, 2018
- Instantons in Self-Organizing Logic GatesPhysical Review Applied, 2018
- Absence of chaos in digital memcomputing machines with solutionsPhysics Letters A, 2017
- Topological Field Theory and Computing with InstantonsAnnalen der Physik, 2017
- Memcomputing Numerical Inversion With Self-Organizing Logic GatesIEEE Transactions on Neural Networks and Learning Systems, 2017
- Polynomial-time solution of prime factorization and NP-complete problems with digital memcomputing machinesChaos: An Interdisciplinary Journal of Nonlinear Science, 2017
- Contrastive divergence for memristor-based restricted Boltzmann machineEngineering Applications of Artificial Intelligence, 2015
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- A Practical Guide to Training Restricted Boltzmann Machines2012
- Learning representations by back-propagating errorsNature, 1986