External validation of a proprietary risk model for 1-year mortality in community-dwelling adults aged 65 years or older
Open Access
- 29 April 2025
- journal article
- Published by Oxford University Press (OUP) in Journal of the American Medical Informatics Association
- Vol. 32 (7) , 1110-1119
Abstract
Objective: To examine the discrimination, calibration, and algorithmic fairness of the Epic End of Life Care Index (EOL-CI). Materials and Methods: We assessed the EOL-CI’s performance by estimating area under the receiver operating characteristic curve (AUC), sensitivity, and positive and negative predictive values in community-dwelling adults ≥65 years of age in a single health system in the Southeastern United States. Algorithmic fairness was examined by comparing the model’s performance across sex, race, and ethnicity subgroups. Using a machine learning approach, we also explored local re-calibration of the EOL-CI considering additional information on past hospitalizations and frailty. Results: Among 215 731 patients (median age = 74 years, 57% female, 12% of Black race), 10% were classified as medium risk (15-44) and 3% as high risk (≥45) by the EOL-CI. The observed 1-year mortality rate was 3%. The EOL-CI had an AUC 0.82 for 1-year mortality, with a positive predictive value of 22%. Predictive performance was generally similar across sex and race subgroups, though the EOL-CI displayed better performance with increasing age and in older adults with 2 or more outpatient encounters in the past 24 months. Local re-calibration of the EOL-CI was required to provide absolute estimates of mortality risk, and calibration was further improved when the EOL-CI was augmented with data on inpatient hospitalizations and frailty. Discussion: The EOL-CI demonstrates reasonable discrimination, albeit with better performance in older adults and in those with greater health system contact. Conclusion: Local refinement and calibration of the EOL-CI score is required to provide direct estimates of prognosis, with the goal of making the EOL-CI a more a valuable tool at the point of care for identifying patients who would benefit from targeted palliative care interventions and proactive care planning.Keywords
Funding Information
- National Institute on Aging
- National Institutes of Health (U54AG063546)
- Imbedded Pragmatic Alzheimer's Disease
- AD-Related Dementias Clinical Trials Collaboratory
- National Center for Advancing Translational Sciences
- National Institutes of Health (UL1TR004929, K23AG070234, K23AG073529)
This publication has 48 references indexed in Scilit:
- AI can be sexist and racist — it’s time to make it fairNature, 2018
- Predictive modeling of U.S. health care spending in late lifeScience, 2018
- Development and Validation of Machine Learning Models for Prediction of 1-Year Mortality Utilizing Electronic Medical Record Data Available at the End of Hospitalization in Multicondition Patients: a Proof-of-Concept StudyJournal of General Internal Medicine, 2018
- Illustrating Informed Presence Bias in Electronic Health Records Data: How Patient Interactions with a Health System Can Impact InferenceeGEMs (Generating Evidence & Methods to improve patient outcomes), 2017
- Change in End-of-Life Care for Medicare BeneficiariesJAMA, 2013
- Analyzing Competing Risk Data Using theRtimeregPackageJournal of Statistical Software, 2011
- Index to Predict 5-Year Mortality of Community-Dwelling Adults Aged 65 and Older Using Data from the National Health Interview SurveyJournal of General Internal Medicine, 2009
- Coding Algorithms for Defining Comorbidities in ICD-9-CM and ICD-10 Administrative DataMedical Care, 2005
- A systematic review of physicians' survival predictions in terminally ill cancer patientsBMJ, 2003
- Extent and determinants of error in doctors' prognoses in terminally ill patients: prospective cohort study Commentary: Why do doctors overestimate? Commentary: Prognoses should be based on proved indices not intuitionBMJ, 2000