Efficient Multiclass ROC Approximation by Decomposition via Confusion Matrix Perturbation Analysis
- 21 March 2008
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 30 (5) , 810-822
Abstract
Receiver operator characteristic (ROC) analysis has become a standard tool in the design and evaluation of two-class classification problems. It allows for an analysis that incorporates all possible priors, costs, and operating points, which is important in many real problems, where conditions are often nonideal. Extending this to the multiclass case is attractive, conferring the benefits of ROC analysis to a multitude of new problems. Even though the ROC analysis extends theoretically to the multiclass case, the exponential computational complexity as a function of the number of classes is restrictive. In this paper, we show that the multiclass ROC can often be simplified considerably because some ROC dimensions are independent of each other. We present an algorithm that analyzes interactions between various ROC dimensions, identifying independent classes, and groups of interacting classes, allowing the ROC to be decomposed. The resulting decomposed ROC hypersurface can be interrogated in a similar fashion to the ideal case, allowing for approaches such as cost-sensitive and Neyman-Pearson optimization, as well as the volume under the ROC. An extensive bouquet of examples and experiments demonstrates the potential of this methodology.Keywords
This publication has 16 references indexed in Scilit:
- Approximating the multiclass ROC by pairwise analysisPattern Recognition Letters, 2007
- The mean subjective utility score, a novel metric for cost-sensitive classifier evaluationPattern Recognition Letters, 2006
- An introduction to ROC analysisPattern Recognition Letters, 2006
- Combining Accuracy and Prior Sensitivity for Classifier Design Under Prior UncertaintyLecture Notes in Computer Science, 2006
- The hypervolume under the ROC hypersurface of "Near-Guessing" and "Near-Perfect" observers in N-class classification tasksIEEE Transactions on Medical Imaging, 2005
- A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification ProblemsMachine Learning, 2001
- Robust Classification for Imprecise EnvironmentsMachine Learning, 2001
- Comparing Three-class Diagnostic Tests by Three-way ROC AnalysisMedical Decision Making, 2000
- Three-way ROCsMedical Decision Making, 1999
- The use of the area under the ROC curve in the evaluation of machine learning algorithmsPattern Recognition, 1997