Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model
Top Cited Papers
- 6 August 2024
- journal article
- research article
- Published by Wiley in Otolaryngology–Head and Neck Surgery
- Vol. 171 (6) , 1751-1757
Abstract
Objective: To use an artificial intelligence (AI)‐powered large language model (LLM) to improve readability of patient handouts.Study Design: Review of online material modified by AI.Setting: Academic center.Methods: Five handout materials obtained from the American Rhinologic Society (ARS) and the American Academy of Facial Plastic and Reconstructive Surgery websites were assessed using validated readability metrics. The handouts were inputted into OpenAI's ChatGPT‐4 after prompting: “Rewrite the following at a 6th‐grade reading level.” The understandability and actionability of both native and LLM‐revised versions were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Results were compared using Wilcoxon rank‐sum tests.Results: The mean readability scores of the standard (ARS, American Academy of Facial Plastic and Reconstructive Surgery) materials corresponded to “difficult,” with reading categories ranging between high school and university grade levels. Conversely, the LLM‐revised handouts had an average seventh‐grade reading level. LLM‐revised handouts had better readability in nearly all metrics tested: Flesch‐Kincaid Reading Ease (70.8 vs 43.9; P < .05), Gunning Fog Score (10.2 vs 14.42; P < .05), Simple Measure of Gobbledygook (9.9 vs 13.1; P < .05), Coleman‐Liau (8.8 vs 12.6; P < .05), and Automated Readability Index (8.2 vs 10.7; P = .06). PEMAT scores were significantly higher in the LLM‐revised handouts for understandability (91 vs 74%; P < .05) with similar actionability (42 vs 34%; P = .15) when compared to the standard materials.Conclusion: Patient‐facing handouts can be augmented by ChatGPT with simple prompting to tailor information with improved readability. This study demonstrates the utility of LLMs to aid in rewriting patient handouts and may serve as a tool to help optimize education materials.Level of Evidence: Level VI.Keywords
This publication has 51 references indexed in Scilit:
- Does ChatGPT Answer Otolaryngology Questions Accurately?The Laryngoscope, 2024
- Dr. Google to Dr. ChatGPT: assessing the content and quality of artificial intelligence-generated medical information on appendicitisSurgical Endoscopy, 2024
- An effort to improve the collection of patient-generated data: readability and understandability of patient-reported outcomes measures in a survivorship cohortQuality of Life Research, 2024
- Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and FormatJAMA Network Open, 2024
- ChatGPT vs. web search for patient questions: what does ChatGPT do better?European Archives of Oto-Rhino-Laryngology, 2024
- Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google BardEuropean Archives of Oto-Rhino-Laryngology, 2023
- Answering head and neck cancer questions: An assessment of ChatGPT responsesAmerican Journal of Otolaryngology, 2023
- Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient InstructionsJAMA Otolaryngology–Head & Neck Surgery, 2023
- Aesthetic Surgery Advice and Counseling from Artificial Intelligence: A Rhinoplasty Consultation with ChatGPTAesthetic Plastic Surgery, 2023
- An evaluation of readability and understandability of online education materials for breast cancer survivorsJournal of Cancer Survivorship, 2022