Leveraging transformers and large language models with antimicrobial prescribing data to predict sources of infection for electronic health record studies
Preprint
Abstract
Background Electronic health records frequently contain extensive unstructured free-text data, but extracting information accurately from these data and at scale is challenging. Using free-text from antibiotic prescribing data as an example, we investigate the performance of modern natural language processing methods (NLP) and large language models (LLMs) as tools for extracting features from medical records. Methods We used 938,150 hospital antibiotic prescriptions from Oxfordshire, UK. The 4000 most frequently used free-text indications justifying antibiotic use were labelled by clinical researchers into 11 categories describing the infection source/clinical syndrome being treated and used for model training. Traditional classification methods, fuzzy regex matching and n-grams with XGBoost, were compared against modern transformer models: we fine-tuned generic and domain-specific BERT models, fine-tuned GPT3.5, and investigated few-shot learning with GPT4. Models were evaluated on internal and external test datasets (2000 prescriptions each). Infection sources determined from ICD10 codes were also used for comparisons. Results In internal and external test datasets, the fine-tuned domain-specific Bio+Clinical BERT model averaged an F1 score of 0.97 and 0.98 respectively across the classes and outperformed the traditional regex (F1=0.71 and 0.74) and n-grams/XGBoost (F1=0.86 and 0.84). OpenAI’s GPT4 model achieved F1 scores of 0.71 and 0.86 without using labelled training data and a fine-tuned GPT3.5 model F1 scores of 0.95 and 0.97. Comparing infection sources extracted from ICD10 codes to those parsed from free-text indications, free-text indications revealed 31% more specific infection sources. Conclusion Modern transformer-based models can efficiently and accurately categorise semi-structured free-text in medical records, such as prescription free-text. Finetuned local transformer models outperform LLMs currently for structured tasks. Few shot LLMs match the performance of traditional NLP without the need for labelling. Transformer-based models have the potential to be used widely throughout medicine to analyse medical records more accurately, facilitating beter research and patient care.Keywords
This publication has 20 references indexed in Scilit:
- Distinct patterns of vital sign and inflammatory marker responses in adults with suspected bloodstream infectionJournal of Infection, 2024
- Large language models encode clinical knowledgeNature, 2023
- Health system-scale language models are all-purpose prediction enginesNature, 2023
- Mortality risks associated with empirical antibiotic activity in Escherichia coli bacteraemia: an analysis of electronic health recordsJournal of Antimicrobial Chemotherapy, 2022
- Inappropriate empirical antibiotic therapy for bloodstream infections based on discordant in-vitro susceptibilities: a retrospective cohort analysis of prevalence, predictors, and mortality risk in US hospitalsThe Lancet Infectious Diseases, 2020
- An evaluation of provider-chosen antibiotic indications as a targeted antimicrobial stewardship interventionAmerican Journal of Infection Control, 2018
- Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009-2014JAMA, 2017
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportabilityJournal of the American Medical Informatics Association, 2016
- NWChem: A comprehensive and scalable open-source solution for large scale molecular simulationsComputer Physics Communications, 2010