Real-World Implementation of Large Language Models for Writing Clinical Discharge Summaries Within a Secure Data Environment: Development and Expert Evaluation Study
Authors:
- Catalina CarenzoCCCatalina Carenzo
- Kathleen GoldsmithKGKathleen Goldsmith
- Maite ArribasMAMaite Arribas
- Benjamin AtkinsBABenjamin Atkins
- Ina KoIKIna Ko
- Ho Lun ChongHCHo Lun Chong
- Asmita RajaARAsmita Raja
- Aya M RiadARAya M Riad
- Rachael LearRLRachael Lear
- Yusuf S AbdullahiYAYusuf S Abdullahi
- Ben GlampsonBGBen Glampson
- Tim OrchardTOTim Orchard
Open Access
- 3 July 2026
- journal article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e88816
Abstract
Background: A discharge summary should be a clinical report that documents a patient’s hospital stay, including test results, diagnoses, management, and follow-up. Currently, discharge summaries are written by clinicians who manually locate pertinent information across the electronic health record, of which approximately 80% is free text. This process is time-consuming and may be suitable for automation using large language models. Objective: This study developed a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the “clinical summary” and “plan and requested actions” sections, from routinely collected electronic patient records. Methods: This study used electronic health record data from Imperial College Healthcare National Health Service Trust, a network of 5 hospitals in northwest London. It was conducted within the Imperial Secure Data Environment under Data Access and Ethics Committee approval. In total, 52 inpatient encounters were selected by the clinical team to ensure diversity in clinical specialty, reason for admission, complexity, length of stay, and sociodemographic characteristics; 83% (n=43) of the cases were allocated to the development dataset, and 17% (n=9) comprised the test dataset. The system synthesized clinical notes related to an inpatient hospital encounter and used structured template prompts with OpenAI’s generative pretrained transformer-4 to generate a discharge summary. The prompt was co-designed across 3 iterations. Resident physicians completed an evaluation form to assess the clinical acceptability of the generated summaries, including the primary outcome (global confidence rating) and secondary outcomes (accuracy, completeness, readability, formatting, sociodemographic bias, and potential clinical harm). Sensitivity analyses assessed the effect of length of stay and admission type (emergency department vs other and surgical vs other) on the primary outcome. Results: A total of 52 patients (n=32, 62% female) were included, with a mean age of 44.8 (SD 27.1) years and an average length of stay of 15.2 (SD 21.1) days. In the test dataset, 89% (8/9) of generative pretrained transformer-4–generated summaries received a positive global confidence rating (“yes” or “yes, with minor changes”). Secondary outcomes were positive for the “clinical summary” section (8/9, 89% complete and 7/9, 78% accurate) and the “plan and requested actions” section (7/9, 78% complete and 7/9, 78% accurate). Readability, formatting, sociodemographic bias, and potential clinical harm also showed positive results in the test dataset. Sensitivity analyses showed no statistically significant variation in the primary outcome across length of stay or admission type (length of stay: P=.29; surgical admission: P=.99; emergency department admission: P=.15). Conclusions: Our results demonstrate the feasibility of the pipeline, but rigorous statistical evaluation in a larger, adequately powered sample is needed.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Large Language Model Assistant for Emergency Department Discharge DocumentationJAMA Network Open, 2025
- Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language ModelJAMA Network Open, 2025
- Physician- and Large Language Model–Generated Hospital Discharge SummariesJAMA Internal Medicine, 2025
- A comparative study of recent large language models on generating hospital discharge summaries for lung cancer patientsJournal of Biomedical Informatics, 2025
- Automated generation of discharge summaries: leveraging large language models with clinical dataScientific Reports, 2025
- Large language models can support generation of standardized discharge summaries – A retrospective study utilizing ChatGPT-4 and electronic health recordsInternational Journal of Medical Informatics, 2024
- Large language models propagate race-based medicinenpj Digital Medicine, 2023
- A method to automate the discharge summary hospital course for neurology patientsJournal of the American Medical Informatics Association, 2023
- Using ChatGPT to write patient clinic lettersThe Lancet Digital Health, 2023
- The Inevitable Application of Big Data to Health CareJAMA, 2013