Using large language models to enhance clinically-driven missing data recovery algorithms in electronic health records
Open Access
- 5 May 2026
- journal article
- Published by Oxford University Press (OUP) in JAMIA Open
- Vol. 9 (3) , ooag080
Abstract
Objectives: Electronic health record (EHR) data are prone to missingness and errors. Previously, we devised an enriched chart review protocol where a “roadmap” of auxiliary diagnoses was used to recover missing values. Still, chart reviews are expensive and time-intensive, limiting the number of patients whose data can be reviewed. Now, we investigate the accuracy and scalability of a roadmap-driven algorithm, based on International Classification of Diseases, 10th revision (ICD-10) codes, to mimic expert chart reviews and recover missing values. Materials and Methods: In addition to the clinicians’ original roadmap from our previous work, we consider new versions that were iteratively refined using large language models (LLMs) in conjunction with clinical expertise to expand the list of auxiliary diagnoses. Using chart reviews for 100 patients from an extensive EHR, we examine algorithm performance. Results: Across 100 chart reviewed patients, there were 413 missing values in the EHR data. The expert chart reviews recovered 49 (12%), while the algorithms using LLMs-enhanced roadmaps recommended almost twice as many (83-89, 20%-22%). The final algorithm using clinician-approved LLMs’ additions offered a balance (73, 18%), expanding the original roadmap with LLMs’ suggestions but only when deemed clinically relevant. When applied to a larger study of 1000 patients from the same EHR, the per-patient median number of non-missing values increased from 6 to 7. Discussion: Clinically-driven algorithms (enhanced by LLMs) can recover missing EHR data with similar accuracy to chart reviews and feasibly be applied to large samples. Extending them to monitor other data quality dimensions is a promising future direction.Keywords
All Related Versions
Funding Information
- Wake Forest University and Wake Forest University School of Medicine (UM1TR004929)
This publication has 74 references indexed in Scilit:
- Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics supportJournal of Biomedical Informatics, 2009
- Combinations of biomarkers predictive of later life mortalityProceedings of the National Academy of Sciences, 2006
- Analysing the hindrance to the use of information and technology for improving efficiency of health care delivery system in TanzaniaTanzania Journal of Health Research, 2006
- Labeling of Medicines and Patient Safety: Evaluating Methods of Reducing Drug Name ConfusionHuman Factors: The Journal of the Human Factors and Ergonomics Society, 2006
- Design and implementation of a health management information system in Malawi: issues, innovations and resultsHealth Policy and Planning, 2005
- The concept of allostasis in biology and biomedicineHormones and Behavior, 2003
- Allostatic load as a marker of cumulative biological risk: MacArthur studies of successful agingProceedings of the National Academy of Sciences, 2001
- Stress, Adaptation, and Disease: Allostasis and Allostatic LoadAnnals of the New York Academy of Sciences, 1998
- Stress and the individual. Mechanisms leading to disease.1993
- International statistical classification of diseases and related health problems. Tenth revision.1988