Three learning stages and accuracy–efficiency tradeoff of restricted Boltzmann machines
Open Access
- 17 September 2022
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 13 (1) , 1-11
Abstract
Restricted Boltzmann Machines (RBMs) offer a versatile architecture for unsupervised machine learning that can in principle approximate any target probability distribution with arbitrary accuracy. However, the RBM model is usually not directly accessible due to its computational complexity, and Markov-chain sampling is invoked to analyze the learned probability distribution. For training and eventual applications, it is thus desirable to have a sampler that is both accurate and efficient. We highlight that these two goals generally compete with each other and cannot be achieved simultaneously. More specifically, we identify and quantitatively characterize three regimes of RBM learning: independent learning, where the accuracy improves without losing efficiency; correlation learning, where higher accuracy entails lower efficiency; and degradation, where both accuracy and efficiency no longer improve or even deteriorate. These findings are based on numerical experiments and heuristic arguments.Keywords
All Related Versions
Funding Information
- MEXT | Japan Society for the Promotion of Science (JP22H01152)
This publication has 51 references indexed in Scilit:
- Learning a Generative Model of Images by Factoring Appearance and ShapeNeural Computation, 2011
- Justifying and Generalizing Contrastive DivergenceNeural Computation, 2009
- Representational Power of Restricted Boltzmann Machines and Deep Belief NetworksNeural Computation, 2008
- To recognize shapes, first learn to generate images2007
- Reducing the Dimensionality of Data with Neural NetworksScience, 2006
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Information geometry on hierarchy of probability distributionsIEEE Transactions on Information Theory, 2001
- Monte Carlo Methods in Statistical Mechanics: Foundations and New Algorithms1997
- Synchronous Boltzmann machines can be universal approximatorsApplied Mathematics Letters, 1996
- The one-dimensional Ising model with a transverse fieldAnnals of Physics, 1970