Glenn Temple Gobbel

BACKGROUND Optimal glycemic control is essential for limiting diabetes–related complications, but the risk of hypoglycemia constrains treatment intensification. Current hypoglycemia surveillance methods relying on diagnosis codes substantially underreport events, and improved methods for identifying hypoglycemia from electronic health records (EHRs) are needed. OBJECTIVE To develop and validate an artificial intelligence (AI) model that can use EHR clinical notes to retrospectively identify severe hypoglycemia events among outpatients with type 2 diabetes (T2D) presenting to the Emergency Department (ED) or requiring an Inpatient Admission (IP). METHODS Using Veterans Health Administration data, we identified a sample of T2D patients in 2019 and classified hypoglycemia via ICD-10 codes or outpatient glucose <70 mg/dL (Hypo+) versus controls (Hypo−). Text corpora included clinical notes near ED encounters (±24 hours) or admission (−12 to +24 hours). We created a hybrid classifier to identify hypoglycemic patients by combining rule-based feature extraction with random forest–based machine learning. A rule–based concept extraction module within the RapTAT natural language processing system generated 44 assertion-aware features. Random forest models were developed and trained using these features and leave-one-out cross-validation and then tested on held-out sets. We created one model for hypoglycemia during ED visits and a second for IP admissions with hypoglycemia. Feature importance was evaluated across both models, and performance was evaluated across different cohorts (2018 vs. 2019) and clinical contexts (presence of low blood glucose <70 mg/dL, hypoglycemia treatment). RESULTS When evaluated across all clinical contexts, the ED model achieved a normalized positive predictive value (PPVN) of 82%-88% and sensitivity of 88%-92%, with specificity of 88% (2018) and 80% (2019). Performance was highest in the context of documented low glucose and hypoglycemia treatment. The IP model performed better overall: PPVN 89%–94%, sensitivity 85%-93%, and specificity 93%-94%. Requiring ≥5 documents improved IP accuracy. Feature importance rankings correlated across models (r=0.74); top predictors included counts and ratios of low glucose phrases and document–level coverage of hypoglycemia concepts. Hypoglycemia cases skewed older, male, and had higher representation of African American patients. CONCLUSIONS AI in the form of a hybrid classification model applied to unstructured EHR notes accurately identified severe hypoglycemia in outpatient T2D populations, outperforming historically reported diagnosis code–based sensitivity and specificity. Models were most effective for admission–related hypoglycemia and when corroborating glucose values and treatment were present. These tools enable scalable surveillance, that may inform safer diabetes management, and support quality improvement.

Background: Despite its high prevalence and clinical impact, research on peripheral artery disease (PAD) remains limited due to poor accuracy of billing codes. Ankle-brachial index (ABI) and toe-brachial index can be used to identify PAD patients with high accuracy within electronic health records. Methods: We developed a novel natural language processing (NLP) algorithm for extracting ABI and toe-brachial index values and laterality (right or left) from ABI reports. A random sample of 800 reports from 94 Veterans Affairs facilities during 2015 to 2017 was selected and annotated by clinical experts. We trained the NLP system using random forest models and optimized it through sequential iterations of 10-fold cross-validation and error analysis on 600 test reports and evaluated its final performance on a separate set of 200 reports. We also assessed the accuracy of NLP-extracted ABI and toe-brachial index values for identifying patients with PAD in a separate cohort undergoing ABI testing. Results: The NLP system had an overall precision (positive predictive value) of 0.85, recall (sensitivity) of 0.93, and F1 measure (accuracy) of 0.89 to correctly identify ABI/toe-brachial index values and laterality. Among 261 patients with ABI testing (49% PAD), the NLP system achieved a positive predictive value of 92.3%, sensitivity of 83.1%, and specificity of 93.1% to identify PAD when compared with a structured chart review. The above findings were consistent in a range of sensitivity analysis. Conclusions: We successfully developed and validated an NLP system for identifying patients with PAD within the Veterans Affairs electronic health record. Our findings have broad implications for PAD research and quality improvement.

Importance Despite its high prevalence and poor outcomes, research on peripheral artery disease (PAD) remains limited due to the poor accuracy of billing codes for identifying PAD in health systems. Objective Design a natural language processing (NLP) system that can extract ankle brachial index (ABI) and toe brachial index (TBI) values and evaluate the performance of extracted ABI/TBI values to identify patients with PAD in the Veterans Health Administration (VHA). Design, Setting, Participants From a corpus of 392,244 ABI test reports at 94 VHA facilities during 2015-2017, we selected a random sample of 800 documents for NLP development. Using machine learning, we designed the NLP system to extract ABI and TBI values and laterality (right or left). Performance was optimized through sequential iterations of 10-fold cross validation and error analysis on 3 sets of 200 documents each, and tested on a final, independent set of 200 documents. Performance of NLP-extracted ABI and TBI values to identify PAD in a random sample of Veterans undergoing ABI testing was compared to structured chart review. Exposure ABI ≤0.9, or TBI ≤0.7 in either right or left limb was used to define PAD at the patient-level Main Outcome Precision (or positive predictive value), recall (or sensitivity), F-1 measure (overall measure of accuracy, defined as harmonic mean of precision and recall) Results The NLP system had an overall precision of 0.85, recall of 0.93 and F1-measure of 0.89 to correctly identify ABI/TBI values and laterality. The F-1 measure was similar for both ABI and TBI (0.88 to 0.91). Recall was higher for ABI (0.95 to 0.97) while precision was higher for TBI (0.94 to 0.95). Among 261 patients with ABI testing (49% with PAD), the NLP system achieved a positive predictive value of 92.3%, sensitivity of 83.1% and specificity of 93.1% to identify PAD when compared to a structured chart review. Conclusion We have successfully developed and validated an NLP system to extract ABI and TBI values which can be used to accurately identify PAD within the VHA. Our findings have broad implications for PAD research and quality improvement efforts in large health systems.

Clinical informatics, decision support for health professionals, electronic health records, and ehealth infrastructures.

BACKGROUND We developed an accurate, stakeholder-informed, automated, natural language processing (NLP) system to measure the quality of heart failure (HF) inpatient care, and explored the potential for adoption of this system within an integrated health care system. OBJECTIVE To accurately automate a United States Department of Veterans Affairs (VA) quality measure for inpatients with HF. METHODS We automated the HF quality measure Congestive Heart Failure Inpatient Measure 19 (CHI19) that identifies whether a given patient has left ventricular ejection fraction (LVEF) <40%, and if so, whether an angiotensin-converting enzyme inhibitor or angiotensin-receptor blocker was prescribed at discharge if there were no contraindications. We used documents from 1083 unique inpatients from eight VA medical centers to develop a reference standard (RS) to train (n=314) and test (n=769) the Congestive Heart Failure Information Extraction Framework (CHIEF). We also conducted semi-structured interviews (n=15) for stakeholder feedback on implementation of the CHIEF. RESULTS The CHIEF classified each hospitalization in the test set with a sensitivity (SN) of 98.9% and positive predictive value of 98.7%, compared with an RS and SN of 98.5% for available External Peer Review Program assessments. Of the 1083 patients available for the NLP system, the CHIEF evaluated and classified 100% of cases. Stakeholders identified potential implementation facilitators and clinical uses of the CHIEF. CONCLUSIONS The CHIEF provided complete data for all patients in the cohort and could potentially improve the efficiency, timeliness, and utility of HF quality measurements.