A Statistical-Modelling Approach to Feedforward Neural Network Model Selection
Preprint
Abstract
Feedforward neural networks (FNNs) can be viewed as non-linear regression models, where covariates enter the model through a combination of weighted summations and non-linear functions. Although these models have some similarities to the approaches used within statistical modelling, the majority of neural network research has been conducted outside of the field of statistics. This has resulted in a lack of statistically-based methodology, and, in particular, there has been little emphasis on model parsimony. Determining the input layer structure is analogous to variable selection, while the structure for the hidden layer relates to model complexity. In practice, neural network model selection is often carried out by comparing models using out-of-sample performance. However, in contrast, the construction of an associated likelihood function opens the door to information-criteria-based variable and architecture selection. A novel model selection method, which performs both input- and hidden-node selection, is proposed using the Bayesian information criterion (BIC) for FNNs. The choice of BIC over out-of-sample performance as the model selection objective function leads to an increased probability of recovering the true model, while parsimoniously achieving favourable out-of-sample performance. Simulation studies are used to evaluate and justify the proposed method, and applications on real data are investigated.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Learning sparse deep neural networks with a spike-and-slab priorStatistics & Probability Letters, 2021
- Bayesian Deep Net GLM and GLMMJournal of Computational and Graphical Statistics, 2019
- State-of-the-art in artificial neural network applications: A surveyHeliyon, 2018
- Variable selection – A review and recommendations for the practicing statisticianBiometrical Journal, 2018
- Design of experiments and focused grid search for neural network parameter optimizationNeurocomputing, 2016
- Information Theory and an Extension of the Maximum Likelihood Principle1998
- Neural Networks and Related Methods for ClassificationJournal of the Royal Statistical Society Series B: Statistical Methodology, 1994
- Network information criterion-determining the number of hidden units for an artificial neural network modelIEEE Transactions on Neural Networks, 1994
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989
- Multilayer feedforward networks are universal approximatorsNeural Networks, 1989