Application of Generative Artificial Intelligence for Physician and Patient Oncology Letters—AI-OncLetters
- 1 May 2025
- journal article
- Published by American Society of Clinical Oncology (ASCO) in JCO Clinical Cancer Informatics
- Vol. 9 (9) , e2400323
Abstract
PURPOSE: Although large language models (LLMs) are increasingly used in clinical practice, formal assessments of their quality, accuracy, and effectiveness in medical oncology remain limited. We aimed to evaluate the ability of ChatGPT, an LLM, to generate physician and patient letters from clinical case notes. METHODS: Six oncologists created 29 (four training, 25 final) synthetic oncology case notes. Structured prompts for ChatGPT were iteratively developed using the four training cases; once finalized, 25 physician-directed and patient-directed letters were generated. These underwent evaluation by expert consumers and oncologists for accuracy, relevance, and readability using Likert scales. The patient letters were also assessed with the Patient Education Materials Assessment Tool for Print (PEMAT-P), Flesch Reading Ease, and Simple Measure of Gobbledygook index. RESULTS: Among physician-to-physician letters, 95% (119/125) of oncologists agreed they were accurate, comprehensive, and relevant, with no safety concerns noted. These letters demonstrated precise documentation of history, investigations, and treatment plans and were logically and concisely structured. Patient-directed letters achieved a mean Flesch Reading Ease score of 73.3 (seventh-grade reading level) and a PEMAT-P score above 80%, indicating high understandability. Consumer reviewers found them clear and appropriate for patient communication. Some omissions of details (eg, side effects), stylistic inconsistencies, and repetitive phrasing were identified, although no clinical safety issues emerged. Seventy-two percent (90/125) of consumers expressed willingness to receive artificial intelligence (AI)-generated patient letters. CONCLUSION: ChatGPT, when guided by structured prompts, can generate high-quality letters that align with clinical and patient communication standards. No clinical safety concerns were identified, although addressing occasional omissions and improving natural language flow could enhance their utility in practice. Further studies comparing AI-generated and human-written letters are recommended.Keywords
This publication has 15 references indexed in Scilit:
- ChatGPT in medicine: A cross-disciplinary systematic review of ChatGPT’s (artificial intelligence) role in research, clinical practice, education, and patient interactionMedicine, 2024
- Generative artificial intelligence in primary care: an online survey of UK general practitionersBMJ Health & Care Informatics, 2024
- The application of large language models in medicine: A scoping reviewiScience, 2024
- Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and FormatJAMA Network Open, 2024
- Using ChatGPT to write patient clinic lettersThe Lancet Digital Health, 2023
- Writing outpatient letters to patientsBMJ, 2020
- Effect of a patient‐directed discharge letter on patient understanding of their hospitalisationInternal Medicine Journal, 2014
- Promoting consultation recording practice in oncology: identification of critical implementation factors and determination of patient benefitPsycho-Oncology, 2012
- The relation between information provision and health-related quality of life, anxiety and depression among cancer survivors: a systematic reviewAnnals of Oncology, 2010
- Oncology service correspondence: Do we communicate?Australasian Radiology, 2003