Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults
Authors:
- Majid AfsharMAMajid Afshar
- Sabrina AdelaineSASabrina Adelaine
- Felice ResnikFRFelice Resnik
- Marlon P MundtMMMarlon P Mundt
- John LongJLJohn Long
- Margaret LeafMLMargaret Leaf
- Theodore AmpianTATheodore Ampian
- Graham J WillsGWGraham J Wills
- Benjamin SchnappBSBenjamin Schnapp
- Michael ChaoMCMichael Chao
- Randy BrownRBRandy Brown
- Cara JoyceCJCara Joyce
Open Access
- 20 April 2023
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 11 (1) , e44977
Abstract
Background: The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. Objective: We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. Methods: The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. Results: The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. Conclusions: The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence–driven CDS. Trial Registration: ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480Keywords
All Related Versions
This publication has 40 references indexed in Scilit:
- Electronic health record adoption in US hospitals: the emergence of a digital “advanced use” divideJournal of the American Medical Informatics Association, 2017
- Readmissions, Observation, and the Hospital Readmissions Reduction ProgramNew England Journal of Medicine, 2016
- A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) projectImplementation Science, 2015
- Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applicationsJournal of the American Medical Informatics Association, 2010
- Assessing the Performance of Prediction ModelsEpidemiology, 2010
- Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation scienceImplementation Science, 2009
- Toyota A3 report: a tool for process improvement in healthcare.2009
- The Basics of Alcohol Screening, Brief Intervention and Referral to Treatment in the Emergency Department2007
- AUDIT‐C as a Brief Screen for Alcohol Misuse in Primary CareAlcohol, Clinical and Experimental Research, 2007
- A comprehensive review of the psychometric properties of the Drug Abuse Screening TestJournal of Substance Abuse Treatment, 2007