Digesting Digital Health: A Study of Appropriateness and Readability of ChatGPT-Generated Gastroenterological Information
Open Access
- 30 August 2024
- journal article
- research article
- Published by Wolters Kluwer Health in Clinical and Translational Gastroenterology
- Vol. 15 (11) , e00765
Abstract
INTRODUCTION: The advent of artificial intelligence–powered large language models capable of generating interactive responses to intricate queries marks a groundbreaking development in how patients access medical information. Our aim was to evaluate the appropriateness and readability of gastroenterological information generated by Chat Generative Pretrained Transformer (ChatGPT). METHODS: We analyzed responses generated by ChatGPT to 16 dialog-based queries assessing symptoms and treatments for gastrointestinal conditions and 13 definition-based queries on prevalent topics in gastroenterology. Three board-certified gastroenterologists evaluated output appropriateness with a 5-point Likert-scale proxy measurement of currency, relevance, accuracy, comprehensiveness, clarity, and urgency/next steps. Outputs with a score of 4 or 5 in all 6 categories were designated as “appropriate.” Output readability was assessed with Flesch Reading Ease score, Flesch-Kinkaid Reading Level, and Simple Measure of Gobbledygook scores. RESULTS: ChatGPT responses to 44% of the 16 dialog-based and 69% of the 13 definition-based questions were deemed appropriate, and the proportion of appropriate responses within the 2 groups of questions was not significantly different (P = 0.17). Notably, none of ChatGPT’s responses to questions related to gastrointestinal emergencies were designated appropriate. The mean readability scores showed that outputs were written at a college-level reading proficiency. DISCUSSION: ChatGPT can produce generally fitting responses to gastroenterological medical queries, but responses were constrained in appropriateness and readability, which limits the current utility of this large language model. Substantial development is essential before these models can be unequivocally endorsed as reliable sources of medical information.Keywords
This publication has 22 references indexed in Scilit:
- Accuracy of ChatGPT in Common Gastrointestinal Diseases: Impact for Patients and ProvidersClinical Gastroenterology and Hepatology, 2023
- Evaluating the performance of ChatGPT in responding to questions about endoscopic procedures for patientsiGIE, 2023
- Accuracy, Reliability, and Comprehensibility of ChatGPT-Generated Medical Responses for Patients With Nonalcoholic Fatty Liver DiseaseClinical Gastroenterology and Hepatology, 2023
- ChatGPT Answers Common Patient Questions About ColonoscopyGastroenterology, 2023
- Appropriateness of Cardiovascular Disease Prevention Recommendations Obtained From a Popular Online Chat-Based Artificial Intelligence ModelJAMA, 2023
- Readability of Patient Education Materials Available at the Point of CareJournal of General Internal Medicine, 2012
- Assessing Readability of Patient Education Materials: Current Role in OrthopaedicsClinical Orthopaedics and Related Research, 2010
- Assessing, Controlling, and Assuring the Quality of Medical Information on the InternetJAMA, 1997
- The use of readability formulas in health carePsychology, Health & Medicine, 1996
- A new readability yardstick.Journal of Applied Psychology, 1948