Feedforward Neural Networks with a Hidden Layer Regularization Method
Open Access
Abstract
In this paper, we propose a group Lasso regularization term as a hidden layer regularization method for feedforward neural networks. Adding a group Lasso regularization term into the standard error function as a hidden layer regularization term is a fruitful approach to eliminate the redundant or unnecessary hidden layer neurons from the feedforward neural network structure. As a comparison, a popular Lasso regularization method is introduced into standard error function of the network. Our novel hidden layer regularization method can force a group of outgoing weights to become smaller during the training process and can eventually be removed after the training process. This means it can simplify the neural network structure and it minimizes the computational cost. Numerical simulations are provided by using K-fold cross-validation method with to avoid overtraining and to select the best learning parameters. The numerical results show that our proposed hidden layer regularization method prunes more redundant hidden layer neurons consistently for each benchmark dataset without loss of accuracy. In contrast, the existing Lasso regularization method prunes only the redundant weights of the network, but it cannot prune any redundant hidden layer neurons.
Keywords
Funding Information
- Natural Science Foundation of China (61473059, 224 11401076 and 61473328)
This publication has 22 references indexed in Scilit:
- A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification ProblemsNeural Processing Letters, 2011
- L 1/2 regularizationScience China Information Sciences, 2010
- AN INCREMENTAL FRAMEWORK BASED ON CROSS-VALIDATION FOR ESTIMATING THE ARCHITECTURE OF A MULTILAYER PERCEPTRONInternational Journal of Pattern Recognition and Artificial Intelligence, 2009
- The Group Lasso for Logistic RegressionJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Model Selection and Estimation in Regression with Grouped VariablesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- The general inefficiency of batch training for gradient descent learningNeural Networks, 2003
- A Penalty-Function Approach for Pruning Feedforward Neural NetworksNeural Computation, 1997
- Artificial neural networks: a tutorialComputer, 1996
- Pruning algorithms-a surveyIEEE Transactions on Neural Networks, 1993
- Learning representations by back-propagating errorsNature, 1986