Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study (Preprint)
Authors:
- Azade TabaieATAzade Tabaie
- Alberta TranATAlberta Tran
- Tony CalabriaTCTony Calabria
- Sonita S BennettSBSonita S Bennett
- Arianna MiliciaAMArianna Milicia
- William WeintraubWWWilliam Weintraub
- William James GallagherWGWilliam James Gallagher
- John YosaitisJYJohn Yosaitis
- Laura C SchubelLSLaura C Schubel
- Mary A HillMHMary A Hill
- Kelly Michelle SmithKSKelly Michelle Smith
- Kristen MillerKMKristen Miller
Preprint
- 17 July 2023
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Diagnostic errors are an underappreciated cause of preventable mortality in hospitals and pose a risk for severe patient harm and increase hospital length of stay. OBJECTIVE This study aims to explore the potential of machine learning and natural language processing techniques in improving diagnostic safety surveillance. We conducted a rigorous evaluation of the feasibility and potential to use electronic health records clinical notes and existing case review data. METHODS Safety Learning System case review data from 1 large health system composed of 10 hospitals in the mid-Atlantic region of the United States from February 2016 to September 2021 were analyzed. The case review outcome included opportunities for improvement including diagnostic opportunities for improvement. To supplement case review data, electronic health record clinical notes were extracted and analyzed. A simple logistic regression model along with 3 forms of logistic regression models (ie, Least Absolute Shrinkage and Selection Operator, Ridge, and Elastic Net) with regularization functions was trained on this data to compare classification performances in classifying patients who experienced diagnostic errors during hospitalization. Further, statistical tests were conducted to find significant differences between female and male patients who experienced diagnostic errors. RESULTS In total, 126 (7.4%) patients (of 1704) had been identified by case reviewers as having experienced at least 1 diagnostic error. Patients who had experienced diagnostic error were grouped by sex: 59 (7.1%) of the 830 women and 67 (7.7%) of the 874 men. Among the patients who experienced a diagnostic error, female patients were older (median 72, IQR 66-80 vs median 67, IQR 57-76; P=.02), had higher rates of being admitted through general or internal medicine (69.5% vs 47.8%; P=.01), lower rates of cardiovascular-related admitted diagnosis (11.9% vs 28.4%; P=.02), and lower rates of being admitted through neurology department (2.3% vs 13.4%; P=.04). The Ridge model achieved the highest area under the receiver operating characteristic curve (0.885), specificity (0.797), positive predictive value (PPV; 0.24), and F1-score (0.369) in classifying patients who were at higher risk of diagnostic errors among hospitalized patients. CONCLUSIONS Our findings demonstrate that natural language processing can be a potential solution to more effectively identifying and selecting potential diagnostic error cases for review and therefore reducing the case review burden.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 26 (1), e50935.
This publication has 28 references indexed in Scilit:
- Predicting misdiagnosed adult-onset type 1 diabetes using machine learningDiabetes Research and Clinical Practice, 2022
- Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timingNature Medicine, 2022
- Harnessing Event Report Data to Identify Diagnostic Error During the COVID-19 PandemicThe Joint Commission Journal on Quality and Patient Safety, 2021
- A predictive model, and predictors of under-five child malaria prevalence in Ghana: How do LASSO, Ridge and Elastic net regression approaches compare?Preventive Medicine Reports, 2021
- Advancing Diagnostic Safety Research: Results of a Systematic Research Priority Setting ExerciseJournal of General Internal Medicine, 2021
- A Scalable Natural Language Processing for Inferring BT-RADS Categorization from Unstructured Brain Magnetic Resonance ReportsJournal of Digital Imaging, 2020
- A new era: artificial intelligence and machine learning in prostate cancerNature Reviews Urology, 2019
- Using Machine Learning Algorithms for Breast Cancer Risk Prediction and DiagnosisProcedia Computer Science, 2016
- Editorial: Helping Health Care Organizations to Define Diagnostic Errors as Missed Opportunities in DiagnosisThe Joint Commission Journal on Quality and Patient Safety, 2014
- Patient Record Review of the Incidence, Consequences, and Causes of Diagnostic Adverse EventsArchives of internal medicine (1960), 2010