Novel Maximum-Margin Training Algorithms for Supervised Neural Networks
- 19 April 2010
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 21 (6) , 972-984
Abstract
This paper proposes three novel training methods, two of them based on the backpropagation approach and a third one based on information theory for multilayer perceptron (MLP) binary classifiers. Both backpropagation methods are based on the maximal-margin (MM) principle. The first one, based on the gradient descent with adaptive learning rate algorithm (GDX) and named maximum-margin GDX (MMGDX), directly increases the margin of the MLP output-layer hyperplane. The proposed method jointly optimizes both MLP layers in a single process, backpropagating the gradient of an MM-based objective function, through the output and hidden layers, in order to create a hidden-layer space that enables a higher margin for the output-layer hyperplane, avoiding the testing of many arbitrary kernels, as occurs in case of support vector machine (SVM) training. The proposed MM-based objective function aims to stretch out the margin to its limit. An objective function based on Lp-norm is also proposed in order to take into account the idea of support vectors, however, overcoming the complexity involved in solving a constrained optimization problem, usually in SVM training. In fact, all the training methods proposed in this paper have time and space complexities O(N) while usual SVM training methods have time complexity O(N 3) and space complexity O(N 2) , where N is the training-data-set size. The second approach, named minimization of interclass interference (MICI), has an objective function inspired on the Fisher discriminant analysis. Such algorithm aims to create an MLP hidden output where the patterns have a desirable statistical distribution. In both training methods, the maximum area under ROC curve (AUC) is applied as stop criterion. The third approach offers a robust training framework able to take the best of each proposed training method. The main idea is to compose a neural model by using neurons extracted from three other neural networks, each one previously trained by MICI, MMGDX, and Levenberg-Marquard (LM), respectively. The resulting neural network was named assembled neural network (ASNN). Benchmark data sets of real-world problems have been used in experiments that enable a comparison with other state-of-the-art classifiers. The results provide evidence of the effectiveness of our methods regarding accuracy, AUC, and balanced error rate.Keywords
This publication has 23 references indexed in Scilit:
- Applications of information theory, genetic algorithms, and neural models to predict oil flowCommunications in Nonlinear Science and Numerical Simulation, 2009
- Multi-class pattern classification using neural networksPattern Recognition, 2007
- Gabor-based kernel pca with fractional power polynomial models for face recognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2004
- Fast obstacle detection for urban traffic situationsIEEE Transactions on Intelligent Transportation Systems, 2002
- Nonlinear kernel-based statistical pattern analysisIEEE Transactions on Neural Networks, 2001
- 10.1162/jmlr.2003.4.6.1071Applied Physics Letters, 2000
- Optimization of neural networks: A comparative analysis of the genetic algorithm and simulated annealingEuropean Journal of Operational Research, 1999
- Exploring constructive cascade networksIEEE Transactions on Neural Networks, 1999
- CARVE-a constructive algorithm for real-valued examplesIEEE Transactions on Neural Networks, 1998
- Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networksNeural Networks, 1990