Updating Clinical Risk Stratification Models Using Rank-Based Compatibility: Approaches for Evaluating and Optimizing Clinician-Model Team Performance
Preprint
Abstract
As data shift or new data become available, updating clinical machine learning models may be necessary to maintain or improve performance over time. However, updating a model can introduce compatibility issues when the behavior of the updated model does not align with user expectations, resulting in poor user-model team performance. Existing compatibility measures depend on model decision thresholds, limiting their applicability in settings where models are used to generate rankings based on estimated risk. To address this limitation, we propose a novel rank-based compatibility measure, $C^R$, and a new loss function that aims to optimize discriminative performance while encouraging good compatibility. Applied to a case study in mortality risk stratification leveraging data from MIMIC, our approach yields more compatible models while maintaining discriminative performance compared to existing model selection techniques, with an increase in $C^R$ of $0.019$ ($95\%$ confidence interval: $0.005$, $0.035$). This work provides new tools to analyze and update risk stratification models used in clinical care.
Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timingNature Medicine, 2022
- The Clinician and Dataset Shift in Artificial IntelligenceNew England Journal of Medicine, 2021
- External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized PatientsJAMA Internal Medicine, 2021
- Multitask learning and benchmarking with clinical time series dataScientific Data, 2019
- Implementation of Dynamically Updated Prediction Models at the Point of Care at a Major Cancer Center: Making Nomograms More Like NetflixUrology, 2016
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessmentIntensive Care Medicine, 2011
- The influence of the sigmoid function parameters on the speed of backpropagation learning1995
- Evaluating the yield of medical testsJAMA, 1982
- A NEW MEASURE OF RANK CORRELATIONBiometrika, 1938