Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest
Preprint
Abstract
The design of AI systems to assist human decision-making typically requires the availability of labels to train and evaluate supervised models. Frequently, however, these labels are unknown, and different ways of estimating them involve unverifiable assumptions or arbitrary choices. In this work, we introduce the concept of label indeterminacy and derive important implications in high-stakes AI-assisted decision-making. We present an empirical study in a healthcare context, focusing specifically on predicting the recovery of comatose patients after resuscitation from cardiac arrest. Our study shows that label indeterminacy can result in models that perform similarly when evaluated on patients with known labels, but vary drastically in their predictions for patients where labels are unknown. After demonstrating crucial ethical implications of label indeterminacy in this high-stakes context, we discuss takeaways for evaluation, reporting, and design.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Recovery Potential in Patients After Cardiac Arrest Who Die After Limitations or Withdrawal of Life SupportJAMA Network Open, 2025
- Beyond model interpretability: socio-structural explanations in machine learningAI & SOCIETY, 2024
- In search of verifiability: Explanations rarely enable complementary performance in AI‐advised decision makingAI Magazine, 2024
- Physicians’ cognitive approach to prognostication after cardiac arrestResuscitation, 2022
- Are providers overconfident in predicting outcome after cardiac arrest?Resuscitation, 2020
- Integration of unsupervised and supervised machine learning algorithms for credit risk assessmentExpert Systems with Applications, 2019
- Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomesResuscitation, 2019
- Delayed awakening after cardiac arrest: prevalence and risk factors in the Parisian registryIntensive Care Medicine, 2016
- Prognostication in comatose survivors of cardiac arrest: An advisory statement from the European Resuscitation Council and the European Society of Intensive Care MedicineIntensive Care Medicine, 2014
- Clinical Nihilism in NeuroemergenciesEmergency Medicine Clinics of North America, 2009