Catalina Carenzo
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Catalina Carenzo has a total of 28 co-authors. Here are the top co-authors by number of shared publications:
- Erik Mayer
Digital Collaboration Space, Imperial Clinical Analytics, Research & Evaluation (iCARE) Secure Data Environment, Imperial College London & Imperial College Healthcare NHS Trust, London, UK
4 shared publications - Asmita Raja
Imperial Clinical Analytics, Research and Evaluation (iCARE), NIHR Imperial Biomedical Research Centre, Imperial College Healthcare NHS Trust, London, England, United Kingdom
3 shared publications - Ben Glampson
Imperial Clinical Analytics, Research and Evaluation (iCARE), NIHR Imperial Biomedical Research Centre, Imperial College Healthcare NHS Trust, London, England, United Kingdom
3 shared publications - Benjamin Atkins
Imperial Clinical Analytics, Research and Evaluation (iCARE), NIHR Imperial Biomedical Research Centre, Imperial College Healthcare NHS Trust, London, England, United Kingdom
3 shared publications - Ho Lun Chong
Imperial Clinical Analytics, Research and Evaluation (iCARE), NIHR Imperial Biomedical Research Centre, Imperial College Healthcare NHS Trust, London, England, United Kingdom
3 shared publications - Ina Ko
Imperial Clinical Analytics, Research and Evaluation (iCARE), NIHR Imperial Biomedical Research Centre, Imperial College Healthcare NHS Trust, London, England, United Kingdom
3 shared publications
Latest Publications
- Open Access
- 3 July 2026
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e88816
Real-World Implementation of Large Language Models for Writing Clinical Discharge Summaries Within a Secure Data Environment: Development and Expert Evaluation Study
Background: A discharge summary should be a clinical report that documents a patient’s hospital stay, including test results, diagnoses, management, and follow-up. Currently, discharge summaries are written by clinicians who manually locate pertinent information across the electronic health record, of which approximately 80% is free text. This process is time-consuming and may be suitable for automation using large language models. Objective: This study developed a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the “clinical summary” and “plan and requested actions” sections, from routinely collected electronic patient records. Methods: This study used electronic health record data from Imperial College Healthcare National Health Service Trust, a network of 5 hospitals in northwest London. It was conducted within the Imperial Secure Data Environment under Data Access and Ethics Committee approval. In total, 52 inpatient encounters were selected by the clinical team to ensure diversity in clinical specialty, reason for admission, complexity, length of stay, and sociodemographic characteristics; 83% (n=43) of the cases were allocated to the development dataset, and 17% (n=9) comprised the test dataset. The system synthesized clinical notes related to an inpatient hospital encounter and used structured template prompts with OpenAI’s generative pretrained transformer-4 to generate a discharge summary. The prompt was co-designed across 3 iterations. Resident physicians completed an evaluation form to assess the clinical acceptability of the generated summaries, including the primary outcome (global confidence rating) and secondary outcomes (accuracy, completeness, readability, formatting, sociodemographic bias, and potential clinical harm). Sensitivity analyses assessed the effect of length of stay and admission type (emergency department vs other and surgical vs other) on the primary outcome. Results: A total of 52 patients (n=32, 62% female) were included, with a mean age of 44.8 (SD 27.1) years and an average length of stay of 15.2 (SD 21.1) days. In the test dataset, 89% (8/9) of generative pretrained transformer-4–generated summaries received a positive global confidence rating (“yes” or “yes, with minor changes”). Secondary outcomes were positive for the “clinical summary” section (8/9, 89% complete and 7/9, 78% accurate) and the “plan and requested actions” section (7/9, 78% complete and 7/9, 78% accurate). Readability, formatting, sociodemographic bias, and potential clinical harm also showed positive results in the test dataset. Sensitivity analyses showed no statistically significant variation in the primary outcome across length of stay or admission type (length of stay: P=.29; surgical admission: P=.99; emergency department admission: P=.15). Conclusions: Our results demonstrate the feasibility of the pipeline, but rigorous statistical evaluation in a larger, adequately powered sample is needed.
- 3 July 2026
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e88816
Open AccessReal-World Implementation of Large Language Models for Writing Clinical Discharge Summaries Within a Secure Data Environment: Development and Expert Evaluation Study
Background: A discharge summary should be a clinical report that documents a patient’s hospital stay, including test results, diagnoses, management, and follow-up. Currently, discharge summaries are written by clinicians who manually locate pertinent information across the electronic health record, of which approximately 80% is free text. This process is time-consuming and may be suitable for automation using large language models. Objective: This study developed a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the “clinical summary” and “plan and requested actions” sections, from routinely collected electronic patient records. Methods: This study used electronic health record data from Imperial College Healthcare National Health Service Trust, a network of 5 hospitals in northwest London. It was conducted within the Imperial Secure Data Environment under Data Access and Ethics Committee approval. In total, 52 inpatient encounters were selected by the clinical team to ensure diversity in clinical specialty, reason for admission, complexity, length of stay, and sociodemographic characteristics; 83% (n=43) of the cases were allocated to the development dataset, and 17% (n=9) comprised the test dataset. The system synthesized clinical notes related to an inpatient hospital encounter and used structured template prompts with OpenAI’s generative pretrained transformer-4 to generate a discharge summary. The prompt was co-designed across 3 iterations. Resident physicians completed an evaluation form to assess the clinical acceptability of the generated summaries, including the primary outcome (global confidence rating) and secondary outcomes (accuracy, completeness, readability, formatting, sociodemographic bias, and potential clinical harm). Sensitivity analyses assessed the effect of length of stay and admission type (emergency department vs other and surgical vs other) on the primary outcome. Results: A total of 52 patients (n=32, 62% female) were included, with a mean age of 44.8 (SD 27.1) years and an average length of stay of 15.2 (SD 21.1) days. In the test dataset, 89% (8/9) of generative pretrained transformer-4–generated summaries received a positive global confidence rating (“yes” or “yes, with minor changes”). Secondary outcomes were positive for the “clinical summary” section (8/9, 89% complete and 7/9, 78% accurate) and the “plan and requested actions” section (7/9, 78% complete and 7/9, 78% accurate). Readability, formatting, sociodemographic bias, and potential clinical harm also showed positive results in the test dataset. Sensitivity analyses showed no statistically significant variation in the primary outcome across length of stay or admission type (length of stay: P=.29; surgical admission: P=.99; emergency department admission: P=.15). Conclusions: Our results demonstrate the feasibility of the pipeline, but rigorous statistical evaluation in a larger, adequately powered sample is needed.
- Open Access
- 13 May 2026
- Published by Elsevier in SSRN Electronic Journal
Large Language Model-Generated Discharge Summaries: A Retrospective Non-Inferiority Study Using Real-World Hospital Data
Background: Hospital discharge summaries are essential for continuity of care, but fragmented electronic health record data can compromise their quality and safety. Large language models (LLMs) are increasingly proposed for clinical documentation tasks, despite limited evidence regarding their performance in healthcare settings. We evaluated the clinical acceptability of GPT-4o-mini-generated discharge summaries derived from electronic health records compared with routine clinician-written summaries. Methods: Retrospective non-inferiority study comparing LLM-generated with clinician-authored discharge summaries, using admission records from five London hospitals (Sept 1, 2022, to Sept 30, 2023). 1294 summaries (645 LLM-generated; 649 clinician-authored) were evaluated by 12 practising doctors. LLM-generated summaries were produced using GPT-4o-mini (2024-07-18) and a clinician co-designed prompt. The primary outcome was clinician willingness to sign the summary with no or minor edits; a 5-percentage-point non-inferiority margin was prespecified. Secondary outcomes assessed accuracy, completeness, tone, clarity, and potential harmfulness of missing or incorrect information on a 5-point Likert scale. Analysis used generalised linear mixed models adjusting for reviewer and case-level variability. Findings: The absolute difference in predicted probability of sign-off (LLM-generated minus clinician-authored) was 22.2 percentage points (95% CI 19.0-27.5) and non-inferiority was demonstrated. Compared with clinician-authored summaries, LLM-generated summaries received higher ratings for accuracy, completeness, tone, and clarity (adjusted risk differences for score ≥4 were 19.4–29.1 percentage points; all 95% CIs excluded zero). LLM-generated discharge summaries had a lower proportion of cases rated ≥3 for potential harm (17.2-19.4%) compared with clinician-authored summaries (37.5-38.5%). Interpretation: LLM-generated discharge summaries meet the current standard of care and represent a promising early use case for generative artificial intelligence (AI) in healthcare with close clinician oversight. Funding This research was funded by the NIHR Imperial Biomedical Research Centre (NIHR203323). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
- 13 May 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessLarge Language Model-Generated Discharge Summaries: A Retrospective Non-Inferiority Study Using Real-World Hospital Data
Background: Hospital discharge summaries are essential for continuity of care, but fragmented electronic health record data can compromise their quality and safety. Large language models (LLMs) are increasingly proposed for clinical documentation tasks, despite limited evidence regarding their performance in healthcare settings. We evaluated the clinical acceptability of GPT-4o-mini-generated discharge summaries derived from electronic health records compared with routine clinician-written summaries. Methods: Retrospective non-inferiority study comparing LLM-generated with clinician-authored discharge summaries, using admission records from five London hospitals (Sept 1, 2022, to Sept 30, 2023). 1294 summaries (645 LLM-generated; 649 clinician-authored) were evaluated by 12 practising doctors. LLM-generated summaries were produced using GPT-4o-mini (2024-07-18) and a clinician co-designed prompt. The primary outcome was clinician willingness to sign the summary with no or minor edits; a 5-percentage-point non-inferiority margin was prespecified. Secondary outcomes assessed accuracy, completeness, tone, clarity, and potential harmfulness of missing or incorrect information on a 5-point Likert scale. Analysis used generalised linear mixed models adjusting for reviewer and case-level variability. Findings: The absolute difference in predicted probability of sign-off (LLM-generated minus clinician-authored) was 22.2 percentage points (95% CI 19.0-27.5) and non-inferiority was demonstrated. Compared with clinician-authored summaries, LLM-generated summaries received higher ratings for accuracy, completeness, tone, and clarity (adjusted risk differences for score ≥4 were 19.4–29.1 percentage points; all 95% CIs excluded zero). LLM-generated discharge summaries had a lower proportion of cases rated ≥3 for potential harm (17.2-19.4%) compared with clinician-authored summaries (37.5-38.5%). Interpretation: LLM-generated discharge summaries meet the current standard of care and represent a promising early use case for generative artificial intelligence (AI) in healthcare with close clinician oversight. Funding This research was funded by the NIHR Imperial Biomedical Research Centre (NIHR203323). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.
- Open Access
- 2 December 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Real-World Implementation of Large Language Models for Writing Clinical Discharge Summaries Within a Secure Data Environment: Development and Expert Evaluation Study (Preprint)
BACKGROUND A discharge summary should be a clinical report that documents a patient’s hospital stay, including test results, diagnoses, management, and follow-up. Currently, discharge summaries are written by clinicians who manually locate pertinent information across the electronic health record, of which approximately 80% is free text. This process is time-consuming and may be suitable for automation using large language models. OBJECTIVE This study developed a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the “clinical summary” and “plan and requested actions” sections, from routinely collected electronic patient records. METHODS This study used electronic health record data from Imperial College Healthcare National Health Service Trust, a network of 5 hospitals in northwest London. It was conducted within the Imperial Secure Data Environment under Data Access and Ethics Committee approval. In total, 52 inpatient encounters were selected by the clinical team to ensure diversity in clinical specialty, reason for admission, complexity, length of stay, and sociodemographic characteristics; 83% (n=43) of the cases were allocated to the development dataset, and 17% (n=9) comprised the test dataset. The system synthesized clinical notes related to an inpatient hospital encounter and used structured template prompts with OpenAI’s generative pretrained transformer-4 to generate a discharge summary. The prompt was co-designed across 3 iterations. Resident physicians completed an evaluation form to assess the clinical acceptability of the generated summaries, including the primary outcome (global confidence rating) and secondary outcomes (accuracy, completeness, readability, formatting, sociodemographic bias, and potential clinical harm). Sensitivity analyses assessed the effect of length of stay and admission type (emergency department vs other and surgical vs other) on the primary outcome. RESULTS A total of 52 patients (n=32, 62% female) were included, with a mean age of 44.8 (SD 27.1) years and an average length of stay of 15.2 (SD 21.1) days. In the test dataset, 89% (8/9) of generative pretrained transformer-4–generated summaries received a positive global confidence rating (“yes” or “yes, with minor changes”). Secondary outcomes were positive for the “clinical summary” section (8/9, 89% complete and 7/9, 78% accurate) and the “plan and requested actions” section (7/9, 78% complete and 7/9, 78% accurate). Readability, formatting, sociodemographic bias, and potential clinical harm also showed positive results in the test dataset. Sensitivity analyses showed no statistically significant variation in the primary outcome across length of stay or admission type (length of stay: P=.29; surgical admission: P=.99; emergency department admission: P=.15). CONCLUSIONS Our results demonstrate the feasibility of the pipeline, but rigorous statistical evaluation in a larger, adequately powered sample is needed.
- 2 December 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessReal-World Implementation of Large Language Models for Writing Clinical Discharge Summaries Within a Secure Data Environment: Development and Expert Evaluation Study (Preprint)
BACKGROUND A discharge summary should be a clinical report that documents a patient’s hospital stay, including test results, diagnoses, management, and follow-up. Currently, discharge summaries are written by clinicians who manually locate pertinent information across the electronic health record, of which approximately 80% is free text. This process is time-consuming and may be suitable for automation using large language models. OBJECTIVE This study developed a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the “clinical summary” and “plan and requested actions” sections, from routinely collected electronic patient records. METHODS This study used electronic health record data from Imperial College Healthcare National Health Service Trust, a network of 5 hospitals in northwest London. It was conducted within the Imperial Secure Data Environment under Data Access and Ethics Committee approval. In total, 52 inpatient encounters were selected by the clinical team to ensure diversity in clinical specialty, reason for admission, complexity, length of stay, and sociodemographic characteristics; 83% (n=43) of the cases were allocated to the development dataset, and 17% (n=9) comprised the test dataset. The system synthesized clinical notes related to an inpatient hospital encounter and used structured template prompts with OpenAI’s generative pretrained transformer-4 to generate a discharge summary. The prompt was co-designed across 3 iterations. Resident physicians completed an evaluation form to assess the clinical acceptability of the generated summaries, including the primary outcome (global confidence rating) and secondary outcomes (accuracy, completeness, readability, formatting, sociodemographic bias, and potential clinical harm). Sensitivity analyses assessed the effect of length of stay and admission type (emergency department vs other and surgical vs other) on the primary outcome. RESULTS A total of 52 patients (n=32, 62% female) were included, with a mean age of 44.8 (SD 27.1) years and an average length of stay of 15.2 (SD 21.1) days. In the test dataset, 89% (8/9) of generative pretrained transformer-4–generated summaries received a positive global confidence rating (“yes” or “yes, with minor changes”). Secondary outcomes were positive for the “clinical summary” section (8/9, 89% complete and 7/9, 78% accurate) and the “plan and requested actions” section (7/9, 78% complete and 7/9, 78% accurate). Readability, formatting, sociodemographic bias, and potential clinical harm also showed positive results in the test dataset. Sensitivity analyses showed no statistically significant variation in the primary outcome across length of stay or admission type (length of stay: P=.29; surgical admission: P=.99; emergency department admission: P=.15). CONCLUSIONS Our results demonstrate the feasibility of the pipeline, but rigorous statistical evaluation in a larger, adequately powered sample is needed.
- Open Access
- 10 July 2025
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 13, e60900
Leveraging AI to Drive Timely Improvements in Patient Experience Feedback: Algorithm Validation
Background: Understanding and improving patient care is pivotal for health care providers. With increasing volumes of the Friends and Family Test (FFT) data in England, manual analysis of this patient feedback poses challenges for many health care organizations. This underscores the importance of automated text analysis, particularly in predicting sentiments and themes in real time. Objective: Leveraging machine learning and natural language processing, this study explores the utility of a supervised algorithm to systematically test and refine the algorithm’s cross-contextual performance in diverse health care settings, addressing variations in population characteristics, geographical locations, and care settings, ultimately driving improvements based on patient feedback. Methods: The text analytics algorithm initially developed in a large acute trust in London was further tested in 9 health care organizations with diverse care settings across England. These trusts varied in technical capacity and resource, population demographics, and FFT free text datasets. Testing and validation of the algorithm were performed, including manual coding of a subset of retrospective comments. Technical infrastructure, including coding environments and packages for algorithm testing and deployment, was optimized. The algorithm was iteratively trained using bag of words from anonymized data, tailored to accommodate contextual variations, and tested for change in algorithm performance while simultaneously rectifying issues identified. Results: The algorithm demonstrated satisfactory overall accuracy (>75%) in predicting themes and sentiments embedded within free-text responses across a variety of care settings and population demographics. While the algorithm yielded strong and reusable models in relatively stable environments, such as adult inpatient care settings, the initial accuracy was notably lower in organizations providing services such as pediatrics and mental health. However, the accuracy of our algorithm significantly improved when individual trust coding templates were applied. Thematic saturation was reached after the fifth organization was recruited, and no further coding was required for the last 4 organizations. Subsequently, a framework and pipeline for deployment of the algorithm were developed to provide a standardized approach for implementation and analysis of FFT free text, ensuring ease of use. Conclusions: This study represents a significant step forward in leveraging free-text FFT data for valuable insights in diverse health care settings through the testing and development of a robust supervised learning text analytics algorithm. The disparity in some care settings was anticipated, given that the lexicon and phraseology used was inherently different from those prevalent in adult inpatient care (where the algorithm was developed). However, these challenges were addressed with further coding and testing. This approach enhanced the accuracy and reliability of the algorithm, encouraged inter- and intraorganizational collaboration, and shared learning.
- 10 July 2025
- Published by JMIR Publications Inc. in JMIR Medical Informatics
- Vol. 13, e60900
Open AccessLeveraging AI to Drive Timely Improvements in Patient Experience Feedback: Algorithm Validation
Background: Understanding and improving patient care is pivotal for health care providers. With increasing volumes of the Friends and Family Test (FFT) data in England, manual analysis of this patient feedback poses challenges for many health care organizations. This underscores the importance of automated text analysis, particularly in predicting sentiments and themes in real time. Objective: Leveraging machine learning and natural language processing, this study explores the utility of a supervised algorithm to systematically test and refine the algorithm’s cross-contextual performance in diverse health care settings, addressing variations in population characteristics, geographical locations, and care settings, ultimately driving improvements based on patient feedback. Methods: The text analytics algorithm initially developed in a large acute trust in London was further tested in 9 health care organizations with diverse care settings across England. These trusts varied in technical capacity and resource, population demographics, and FFT free text datasets. Testing and validation of the algorithm were performed, including manual coding of a subset of retrospective comments. Technical infrastructure, including coding environments and packages for algorithm testing and deployment, was optimized. The algorithm was iteratively trained using bag of words from anonymized data, tailored to accommodate contextual variations, and tested for change in algorithm performance while simultaneously rectifying issues identified. Results: The algorithm demonstrated satisfactory overall accuracy (>75%) in predicting themes and sentiments embedded within free-text responses across a variety of care settings and population demographics. While the algorithm yielded strong and reusable models in relatively stable environments, such as adult inpatient care settings, the initial accuracy was notably lower in organizations providing services such as pediatrics and mental health. However, the accuracy of our algorithm significantly improved when individual trust coding templates were applied. Thematic saturation was reached after the fifth organization was recruited, and no further coding was required for the last 4 organizations. Subsequently, a framework and pipeline for deployment of the algorithm were developed to provide a standardized approach for implementation and analysis of FFT free text, ensuring ease of use. Conclusions: This study represents a significant step forward in leveraging free-text FFT data for valuable insights in diverse health care settings through the testing and development of a robust supervised learning text analytics algorithm. The disparity in some care settings was anticipated, given that the lexicon and phraseology used was inherently different from those prevalent in adult inpatient care (where the algorithm was developed). However, these challenges were addressed with further coding and testing. This approach enhanced the accuracy and reliability of the algorithm, encouraged inter- and intraorganizational collaboration, and shared learning.