Average Contrastive Divergence for Training Restricted Boltzmann Machines
Open Access
Abstract
This paper studies contrastive divergence (CD) learning algorithm and proposes a new algorithm for training restricted Boltzmann machines (RBMs). We derive that CD is a biased estimator of the log-likelihood gradient method and make an analysis of the bias. Meanwhile, we propose a new learning algorithm called average contrastive divergence (ACD) for training RBMs. It is an improved CD algorithm, and it is different from the traditional CD algorithm. Finally, we obtain some experimental results. The results show that the new algorithm is a better approximation of the log-likelihood gradient method and outperforms the traditional CD algorithm.Keywords
Funding Information
- the National Science Foundation of China (61273365, 11407776)
- National High Technology Research and Development Program of China (2012AA011103)
This publication has 16 references indexed in Scilit:
- Convergence Analysis of Contrastive Divergence Algorithm Based on Gradient Method with ErrorsMathematical Problems in Engineering, 2015
- Deep correspondence restricted Boltzmann machine for cross-modal retrievalNeurocomputing, 2015
- Improving mixing rate with tempered transition for learning restricted Boltzmann machinesNeurocomputing, 2014
- Enhanced Gradient for Training Restricted Boltzmann MachinesNeural Computation, 2013
- Bounding the Bias of Contrastive Divergence LearningNeural Computation, 2011
- Justifying and Generalizing Contrastive DivergenceNeural Computation, 2009
- Learning multiple layers of representationTrends in Cognitive Sciences, 2007
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Parametric Inference for imperfectly observed Gibbsian fieldsProbability Theory and Related Fields, 1989