Accelerating Deep Learning with Memcomputing
Authors:
- Haik ManukianHMHaik Manukian
- Fabio L. TraversaFTFabio L. Traversa
- Massimiliano Di VentraMVMassimiliano Di Ventra
Preprint
Abstract
Restricted Boltzmann machines (RBMs) and their extensions, often called "deep-belief networks", are very powerful neural networks that have found widespread applicability in the fields of machine learning and big data. The standard way to training these models resorts to an iterative unsupervised procedure based on Gibbs sampling, called "contrastive divergence", and additional supervised tuning via back-propagation. However, this procedure has been shown not to follow any gradient and can lead to suboptimal solutions. In this paper, we show a very efficient alternative to contrastive divergence by means of simulations of digital memcomputing machines (DMMs). We test our approach on pattern recognition using the standard MNIST data set of hand-written numbers. DMMs sample very effectively the vast phase space defined by the probability distribution of RBMs over the test sample inputs, and provide a very good approximation close to the optimum. This efficient search significantly reduces the number of generative pre-training iterations necessary to achieve a given level of accuracy in the MNIST data set, as well as a total performance gain over the traditional approaches. In fact, the acceleration of the pre-training achieved by simulating DMMs is comparable to, in number of iterations, the recently reported hardware application of the quantum annealing method on the same network and data set. Notably, however, DMMs perform far better than the reported quantum annealing results in terms of quality of the training. Our approach is agnostic about the connectivity of the network. Therefore, it can be extended to train full Boltzmann machines, and even deep networks at once.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Evidence of Exponential Speed‐Up in the Solution of Hard Optimization ProblemsComplexity, 2018
- Absence of chaos in digital memcomputing machines with solutionsPhysics Letters A, 2017
- Topological Field Theory and Computing with InstantonsAnnalen der Physik, 2017
- Memcomputing Numerical Inversion With Self-Organizing Logic GatesIEEE Transactions on Neural Networks and Learning Systems, 2017
- Polynomial-time solution of prime factorization and NP-complete problems with digital memcomputing machinesChaos: An Interdisciplinary Journal of Nonlinear Science, 2017
- Contrastive divergence for memristor-based restricted Boltzmann machineEngineering Applications of Artificial Intelligence, 2015
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- A Practical Guide to Training Restricted Boltzmann Machines2012
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Learning representations by back-propagating errorsNature, 1986