Leveraging Large Language Models to Develop an Interpretable Prediction Model for Postpartum Hemorrhage Prior to the Onset of Labor
Preprint
Abstract
Objective: To evaluate whether large language models (LLMs) applied to prenatal clinical notes can predict postpartum hemorrhage (PPH) prior to the onset of labor and to compare model performance across outcome definitions, including a novel intervention-based definition. Methods: We conducted a retrospective cohort study of 19,992 deliveries within a large regional health network. Two outcome definitions for PPH were used: estimated or quantitative blood loss (EBL/QBL) extracted from clinical notes, and a clinical intervention-based definition (cPPH) incorporating transfusion, uterotonics, Bakri balloon, or hysterectomy. We evaluated three approaches for PPH prediction: (1) supervised machine learning using structured electronic medical record data; (2) direct prediction using a fine-tuned LLM applied to clinical notes; and (3) interpretable models using LLM-extracted features combined with structured data. Model performance was evaluated using area under the receiver operating characteristic curve (AUROC) on a temporally held-out test set. Results: The LLM-based direct prediction model achieved the highest performance for both PPH definitions (AUROC 0.79–0.80), followed by interpretable models combining LLM-extracted features with structured data (AUROC 0.76–0.78). Models using only structured data performed worse (AUROC 0.65–0.71). The LLM-extracted features approach identified 47 significant predictors, including established risk factors such as multiple gestation and previous cesarean delivery. Demographic differences were observed between PPH definitions: mothers who met only the cPPH definition had lower gestational age and higher rates of cesarean delivery compared to those meeting only the EBL/QBL definition. Conclusion: These findings highlight the potential of LLM-based approaches for enhancing PPH risk stratification, with the feature extraction method offering a promising balance between predictive performance and clinical utility. Integrating these methods into clinical workflows could improve early detection and guide targeted preventive interventions.Keywords
This publication has 25 references indexed in Scilit:
- Prediction of post-delivery hemoglobin levels with machine learning algorithmsScientific Reports, 2024
- Zero-shot interpretable phenotyping of postpartum hemorrhage using large language modelsnpj Digital Medicine, 2023
- Predicting seizure recurrence after an initial seizure-like episode from routine clinical notes using large language models: a retrospective cohort studyThe Lancet Digital Health, 2023
- Health system-scale language models are all-purpose prediction enginesNature, 2023
- Machine learning-based prediction of postpartum hemorrhage after vaginal delivery: combining bleeding high risk factors and uterine contraction curveArchives of Gynecology and Obstetrics, 2022
- Postpartum hemorrhage risk is driven by changes in blood composition through pregnancyScientific Reports, 2021
- Machine learning for patient risk stratification: standing on, or looking over, the shoulders of clinicians?npj Digital Medicine, 2021
- Maternal and neonatal complications of fetal macrosomia: cohort studyUltrasound in Obstetrics & Gynecology, 2019
- Postpartum hemorrhage: new insights for definition and diagnosisAmerican Journal of Obstetrics and Gynecology, 2018
- Global causes of maternal death: a WHO systematic analysisThe Lancet Global Health, 2014