Area under the ROC Curve has the most consistent evaluation for binary classification
Author:
Top Cited Papers
Open Access
- 23 December 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (12) , e0316019
Abstract
The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables and the sample size are kept constant. Analyzing 156 data scenarios, 18 model evaluation metrics and five commonly used machine learning models as well as a naive random guess model, I find that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, Area Under the ROC Curve (AUC) which takes all decision thresholds into account when evaluating models has the smallest variance in evaluating individual models and smallest variance in ranking of a set of models. A close threshold analysis using all possible thresholds for all metrics further supports the hypothesis that considering all decision thresholds helps reduce the variance in model evaluation with respect to prevalence change in data. The results have significant implications for model evaluation and model selection in binary classification tasks.Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scoresThe Journal of Thoracic and Cardiovascular Surgery, 2023
- General Performance Score for classification problemsApplied Intelligence, 2022
- Unified Performance Measure for Binary Classification Problems2020
- On the assessment of software defect prediction models via ROC curvesEmpirical Software Engineering, 2020
- A novel weighted TPR-TNR measure to assess performance of the classifiersExpert Systems with Applications, 2020
- ROC and AUC with a Binary Predictor: a Potentially Misleading MetricJournal of Classification, 2019
- Rethinking an ROC partial area index for evaluating the classification performance at a high specificity rangeAdvances in Data Analysis and Classification, 2017
- The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseasesJournal of Clinical Epidemiology, 2015
- Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approachEuropean Radiology, 2015
- Bayes' Rule2015