Convergence of Gradient Method With Momentum for Two-Layer Feedforward Neural Networks
Authors:
- Wei WuWWWei Wu
- Gaofeng ZhengGZGaofeng Zheng
- 6 March 2006
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 17 (2) , 522-525
Abstract
A gradient method with momentum for two-layer feedforward neural networks is considered. The learning rate is set to be a constant and the momentum factor an adaptive variable. Both the weak and strong convergence results are proved, as well as the convergence rates for the error function and for the weight. Compared to the existing convergence results, our results are more general since we do not require the error function to be quadratic.Keywords
This publication has 10 references indexed in Scilit:
- Convergence of an online gradient method for feedforward neural networks with stochastic inputsJournal of Computational and Applied Mathematics, 2004
- Steepest descent with momentum for quadratic functions is a version of the conjugate gradient methodNeural Networks, 2004
- Training Multilayer Perceptrons Via Minimization of Sum of Ridge FunctionsAdvances in Computational Mathematics, 2002
- Stability of steepest descent with momentum for quadratic functionsIEEE Transactions on Neural Networks, 2002
- Deterministic convergence of an online gradient method for neural networksJournal of Computational and Applied Mathematics, 2002
- Parameter Convergence and Learning Curves for Neural NetworksNeural Computation, 1999
- Diffusion Approximations for the Constant Learning Rate Backpropagation Algorithm and Resistance to Local MinimaNeural Computation, 1994
- On the Convergence of the LMS Algorithm with Adaptive Learning Rate for Linear Feedforward NetworksNeural Computation, 1991
- An adaptive training algorithm for back propagation networksComputer Speech & Language, 1987
- Learning representations by back-propagating errorsNature, 1986