Leveraging a large language model to support expansion of surveillance activities to include cardiovascular implantable device infections in a large, integrated national healthcare system
Open Access
- 23 January 2026
- journal article
- research article
- Published by Cambridge University Press (CUP) in Infection Control & Hospital Epidemiology
- Vol. 47 (4) , 351-356
Abstract
Background: Surveillance activities are emerging as exemplar use cases for large language models (LLMs) in health care. The aim of this study was to evaluate the potential for LLMs to support the expansion of surveillance activities to include cardiovascular implantable electronic device (CIED) procedures. Methods: A validated machine learning-based infection flagging tool was applied to a cohort of VA CIED procedures from 7/1/2021 to 9/30/2023; cases with ≥10% probability of CIED infection underwent manual review. Then, a weighted random sample of 50 infected and 50 uninfected cases was reviewed with generative artificial intelligence (GenAI) assistance. GenAI prompts were iteratively refined to extract and classify all components of infection-related variables from clinical notes. Data extracted by GenAI were compared with manual chart reviews to assess infection status and extraction consistency. Results: Among 12,927 CIED procedures, 334 (2.58%) had ≥10% probability of CIED infection. Among 100 sampled cases, 50 of 50 uninfected cases were correctly categorized. Among 50 infection cases, GenAI identified all CIED infections, but the timing of events and the attribution to a preceding procedure were incorrect in 7 of 50 cases. The overall specificity of the GenAI-assisted process was 100% and the sensitivity for accurately classifying timing and attribution of CIED infection events was 82%. Errors in timing improved with iterative prompt updates. Manual chart reviews averaged 25 minutes per chart; the GenAI-assisted process averaged 5–7 minutes per chart. Conclusions: LLMs can help streamline the review process for healthcare-associated infection surveillance, but manual adjudication of output is needed to ensure the correct timeline of events and attribution.Keywords
This publication has 18 references indexed in Scilit:
- Physician- and Large Language Model–Generated Hospital Discharge SummariesJAMA Internal Medicine, 2025
- Time to Study Implementation of AI-Generated Discharge SummariesJAMA Internal Medicine, 2025
- Cardiac Implantable Device Infection Surveillance AlgorithmJAMA Network Open, 2025
- A comprehensive survey on integrating large language models with knowledge-based methodsKnowledge-Based Systems, 2025
- Performance of a large language model for identifying central line-associated bloodstream infections (CLABSI) using real clinical notesInfection Control & Hospital Epidemiology, 2024
- ChatGPT: Increasing accessibility for natural language processing in healthcare quality measurementInfection Control & Hospital Epidemiology, 2023
- Application of OpenAI GPT-4 for the retrospective detection of catheter-associated urinary tract infections in a fictitious and curated patient data setInfection Control & Hospital Epidemiology, 2023
- Leveraging electronic data to expand infection detection beyond traditional settings and definitions (Part II/III)Antimicrobial Stewardship & Healthcare Epidemiology, 2023
- Automating surveillance for healthcare-associated infections: Rationale and current realities (Part I/III)Antimicrobial Stewardship & Healthcare Epidemiology, 2023
- Novel Method to Flag Cardiac Implantable Device Infections by Integrating Text Mining With Structured Data in the Veterans Health Administration’s Electronic Medical RecordJAMA Network Open, 2020