Convergence Analysis of Contrastive Divergence Algorithm Based on Gradient Method with Errors
Authors:
Open Access
- 12 July 2015
- journal article
- research article
- Published by Wiley in Mathematical Problems in Engineering
- Vol. 2015, 1-9
Abstract
Contrastive Divergence has become a common way to train Restricted Boltzmann Machines; however, its convergence has not been made clear yet. This paper studies the convergence of Contrastive Divergence algorithm. We relate Contrastive Divergence algorithm to gradient method with errors and derive convergence conditions of Contrastive Divergence algorithm using the convergence theorem of gradient method with errors. We give specific convergence conditions of Contrastive Divergence learning algorithm for Restricted Boltzmann Machines in which both visible units and hidden units can only take a finite number of values. Two new convergence conditions are obtained by specifying the learning rate. Finally, we give specific conditions that the step number of Gibbs sampling must be satisfied in order to guarantee the Contrastive Divergence algorithm convergence.Keywords
Funding Information
- National Natural Science Foundation of China (61273365, 11407776, 2012AA011103)
- National Natural Science Foundation of China (61273365, 11407776, 2012AA011103)
- National High Technology Research and Development Program of China (61273365, 11407776, 2012AA011103)
This publication has 11 references indexed in Scilit:
- Deep correspondence restricted Boltzmann machine for cross-modal retrievalNeurocomputing, 2015
- Bounding the Bias of Contrastive Divergence LearningNeural Computation, 2011
- Acoustic Modeling Using Deep Belief NetworksIEEE Transactions on Audio, Speech, and Language Processing, 2011
- Justifying and Generalizing Contrastive DivergenceNeural Computation, 2009
- Reducing the Dimensionality of Data with Neural NetworksScience, 2006
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Gradient Convergence in Gradient methods with ErrorsSIAM Journal on Optimization, 2000
- On the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity ratesStochastics and Stochastic Reports, 1999
- Asynchronous Stochastic ApproximationsSIAM Journal on Control and Optimization, 1998
- A Stochastic Approximation MethodThe Annals of Mathematical Statistics, 1951