Beyond Area Under the Receiver Operating Characteristic Curve: Evaluating Predictive Performance Metrics Under Class Imbalance in Real-World Clinical Data
Authors:
- Vanessa das Graças José VenturaVVVanessa das Graças José Ventura
- Claudio Moisés Valiense de AndradeCAClaudio Moisés Valiense de Andrade
- Jussara Marques de AlmeidaJAJussara Marques de Almeida
- Bruno Porto PessoaBPBruno Porto Pessoa
- Carísi Anne PolanczykCPCarísi Anne Polanczyk
- Guilherme Fonseca Do NascimentoGNGuilherme Fonseca Do Nascimento
- Eric BoersmaEBEric Boersma
- Heloisa Reniers ViannaHVHeloisa Reniers Vianna
- Katia de Paula FarahKFKatia de Paula Farah
- Leonardo Chaves Dutra da RochaLRLeonardo Chaves Dutra da Rocha
- Marcos André GonçalvesMGMarcos André Gonçalves
- Milena Soriano MarcolinoMMMilena Soriano Marcolino
Open Access
- 24 June 2026
- journal article
- machine learning
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e86379
Abstract
Background: Predictive models increasingly support clinical decision-making, although imbalanced outcome distributions are common in health care datasets and can distort performance evaluation. The area under the receiver operating characteristic curve (AUROC) remains the most frequently reported metric, despite its limited ability to reflect clinically meaningful performance under class imbalance. Objective: This study aimed to examine the influences of metric selection on the clinical interpretation of predictive models in imbalanced real-world health care data. Methods: This was a retrospective cohort study, including 17,018 hospitalized patients with COVID-19. Two predictive models using extreme gradient boosting (XGBoost) were developed to predict kidney replacement therapy (KRT) and mortality. Model performance was assessed using AUROC, macro- F 1 -score, class-specific precision and recall, calibration (curve, slope, and intercept), decision curve analysis, and learning curves. Standard rebalancing strategies were applied exclusively to the training data to evaluate their impact on performance. Results: KRT occurred in 9.5%, and mortality in 18.0%. Although AUROC values were high (0.928 for KRT and 0.945 for mortality), performance in the minority class was substantially lower. For KRT, precision was 0.539 and recall 0.372; for mortality, precision was 0.725 and recall 0.718. Rebalancing strategies were associated with higher recall for the minority class, but this gain was accompanied by a reduction in precision, with minimal impact on AUROC values. As a result, AUROC remained high despite clinically relevant changes in error distribution between false positives and false negatives. The learning curves show a plateau-like shape, with stable validation performance across all training set sizes for both outcomes. Conclusions: AUROC alone is insufficient to evaluate prediction models in imbalanced health care scenarios, even with rebalancing. Routine reporting of class-aware metrics, alongside learning curve analysis, is essential to support robust and clinically meaningful evaluation of predictive models, rather than their direct translation into practice.Keywords
This publication has 63 references indexed in Scilit:
- The Harms of Class Imbalance Corrections for Machine Learning Based Prediction Models: A Simulation StudyStatistics in Medicine, 2025
- A practical guide to the implementation of AI in orthopaedic research, Part 6: How to evaluate the performance of AI research?Journal of Experimental Orthopaedics, 2024
- Artificial Intelligence and Machine Learning in Health Care and Medical Sciences2024
- Behavioral Counseling Interventions to Promote a Healthy Diet and Physical Activity for Cardiovascular Disease Prevention in Adults With Cardiovascular Risk FactorsJAMA, 2020
- Clinical Prediction Models2019
- Data Science and Predictive Analytics2018
- Discrimination and Calibration of Clinical Prediction ModelsJAMA, 2017
- Optimal Thresholding of Classifiers to Maximize F1 Measure2014
- Applied Logistic Regression2013
- Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning2005