Optimized Approximation Algorithm in Neural Networks Without Overfitting
Authors:
- Yinyin LiuYLYinyin Liu
- Janusz A. StarzykJSJanusz A. Starzyk
- Zhen ZhuZZZhen Zhu
- 14 March 2008
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 19 (6) , 983-995
Abstract
In this paper, an optimized approximation algorithm (OAA) is proposed to address the overfitting problem in function approximation using neural networks (NNs). The optimized approximation algorithm avoids overfitting by means of a novel and effective stopping criterion based on the estimation of the signal-to-noise-ratio figure (SNRF). Using SNRF, which checks the goodness-of-fit in the approximation, overfitting can be automatically detected from the training error only without use of a separate validation set. The algorithm has been applied to problems of optimizing the number of hidden neurons in a multilayer perceptron (MLP) and optimizing the number of learning epochs in MLP's backpropagation training using both synthetic and benchmark data sets. The OAA algorithm can also be utilized in the optimization of other parameters of NNs. In addition, it can be applied to the problem of function approximation using any kind of basis functions, or to the problem of learning model selection when overfitting needs to be considered.Keywords
This publication has 27 references indexed in Scilit:
- Linear-Least-Squares Initialization of Multilayer Perceptrons Through Backpropagation of the Desired ResponseIEEE Transactions on Neural Networks, 2005
- Learning capability and storage capacity of two-hidden-layer feedforward networksIEEE Transactions on Neural Networks, 2003
- Specification of Training Sets and the Number of Hidden Neurons for Multilayer PerceptronsNeural Computation, 2001
- Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functionsIEEE Transactions on Neural Networks, 1998
- Network information criterion-determining the number of hidden units for an artificial neural network modelIEEE Transactions on Neural Networks, 1994
- A fuzzy-logic-based approach to qualitative modelingIEEE Transactions on Fuzzy Systems, 1993
- An information criterion for optimal neural network selectionIEEE Transactions on Neural Networks, 1991
- Stochastic Complexity and ModelingThe Annals of Statistics, 1986
- Oscillation and Chaos in Physiological Control SystemsScience, 1977
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974