AUC-Maximizing Ensembles through Metalearning
Authors:
Open Access
- 8 December 2015
- journal article
- research article
- Published by Walter de Gruyter GmbH in The International Journal of Biostatistics
- Vol. 12 (1) , 203-218
Abstract
Area Under the ROC Curve (AUC) is often used to measure the performance of an estimator in binary classification problems. An AUC-maximizing classifier can have significant advantages in cases where ranking correctness is valued or if the outcome is rare. In a Super Learner ensemble, maximization of the AUC can be achieved by the use of an AUC-maximining metalearning algorithm. We discuss an implementation of an AUC-maximization technique that is formulated as a nonlinear optimization problem. We also evaluate the effectiveness of a large number of different nonlinear optimization algorithms to maximize the cross-validated AUC of the ensemble fit. The results provide evidence that AUC-maximizing metalearners can, and often do, out-perform non-AUC-maximizing metalearning methods, with respect to ensemble AUC. The results also demonstrate that as the level of imbalance in the training data increases, the Super Learner ensemble outperforms the top base algorithm by a larger degree.Keywords
This publication has 25 references indexed in Scilit:
- Super LearnerStatistical Applications in Genetics and Molecular Biology, 2007
- Algorithm 778: L-BFGS-BACM Transactions on Mathematical Software, 1997
- The use of the area under the ROC curve in the evaluation of machine learning algorithmsPattern Recognition, 1997
- Convergence theorems for a class of simulated annealing algorithms on ℝdJournal of Applied Probability, 1992
- Analyzing a Portion of the ROC CurveMedical Decision Making, 1989
- Information retrievalACM SIGIR Forum, 1983
- The Convergence of a Class of Double-rank Minimization Algorithms 1. General ConsiderationsIMA Journal of Applied Mathematics, 1970
- A new approach to variable metric algorithmsThe Computer Journal, 1970
- A family of variable-metric methods derived by variational meansMathematics of Computation, 1970
- Conditioning of quasi-Newton methods for function minimizationMathematics of Computation, 1970