Hybrid rule-based and on-premises LLM pipeline for extracting CMR and CPET metrics from free-text reports in repaired tetralogy of Fallot
Authors:
- Izzet Turkalp AkbasliIAIzzet Turkalp Akbasli
- Kristen L. BeckKristen L. Beck
- Wilson LiouWLWilson Liou
- Sherry DuSDSherry Du
- Gabriel NemerGNGabriel Nemer
- Orkun BalogluOBOrkun Baloglu
- Samir Q. LatifiSLSamir Q. Latifi
- Bradley S. MarinoBMBradley S. Marino
- Samer AlbahraSASamer Albahra
- Animesh TandonATAnimesh Tandon
Preprint
Abstract
Background: Patients with repaired tetralogy of Fallot (rTOF) require lifelong surveillance with cardiovascular magnetic resonance (CMR) and cardiopulmonary exercise testing (CPET). However, results are frequently stored as unstructured free-text reports, hindering large-scale analysis and research. Objectives: This study aimed to develop and evaluate a privacy-preserving hybrid natural language processing pipeline combining regular expressions (regex) and an on-premises large language model (LLM) to accurately extract key CMR and CPET metrics from legacy free-text reports in patients with rTOF. Methods: We retrospectively analyzed 430 CMR and 262 CPET reports (2005–2023) from patients with rTOF. A two-stage hybrid pipeline was implemented: regex rules were applied first, followed by targeted prompting of an on-premises Llama-3.1-8B-Instruct LLM only when regex failed or returned ambiguous results. Performance was compared against regex-only and LLM-only approaches using coverage, precision, recall, and F1-score. Results: In CMR reports, the hybrid pipeline achieved perfect coverage (1.00) and F1-score (1.00) versus 0.98 coverage and 0.93 F1-score with regex alone, while reducing computational cost by ~75% compared with LLM-only. In CPET reports, the hybrid approach improved F1-score from 0.74 (regex alone) to 0.98, with particularly large gains for semantically complex variables (e.g., peak VO □, test termination reason). Conclusions: A hybrid regex–on-premises LLM pipeline provides near-perfect, efficient, and HIPAA-compliant extraction of clinical metrics from unstructured cardiology reports, offering a scalable solution for retrospective research and quality improvement in congenital heart disease.Keywords
This publication has 37 references indexed in Scilit:
- Medical language model specialized in extracting cardiac knowledgeScientific Reports, 2024
- Society for Cardiovascular Magnetic Resonance guidelines for reporting cardiovascular magnetic resonance examinations in patients with congenital heart diseaseJournal of Cardiovascular Magnetic Resonance, 2024
- A review of green artificial intelligence: Towards a more sustainable futureNeurocomputing, 2024
- Recent methodological advances in federated learning for healthcarePatterns, 2024
- An in-depth evaluation of federated learning on biomedical natural language processing for information extractionnpj Digital Medicine, 2024
- AI Chatbots, Health Privacy, and Challenges to HIPAA ComplianceJAMA, 2023
- Radiology in the era of large language models: the near and the dark side of the moonEuropean Radiology, 2023
- Ethics of large language models in medicine and medical researchThe Lancet Digital Health, 2023
- BERT-based natural language processing analysis of French CT reports: Application to the measurement of the positivity rate for pulmonary embolismResearch in Diagnostic and Interventional Imaging, 2023
- Cardiovascular magnetic resonance in the follow-up of patients with corrected tetralogy of Fallot: A reviewAmerican Heart Journal, 2006