Comprehensibility and readability of selected artificial intelligence chatbots in providing uveitis-related information
Open Access
- 24 October 2025
- journal article
- research article
- Published by Wolters Kluwer Health in Medicine
- Vol. 104 (43) , e45135
Abstract
This study aims to evaluate and compare the quality and comprehensibility of responses generated by 5 artificial intelligence chatbots – ChatGPT-4, Claude, Mistral, Grok, and Google PaLM – to the most frequently asked questions about uveitis. Google Trends was employed to identify significant phrases associated with uveitis. Each artificial intelligence chatbot was provided with a unique sequence of 25 frequently searched terms as input. The responses were evaluated using 3 distinct tools: The Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P), the Simple Measure of Gobbledygook (SMOG) index, and the Automated Readability Index (ARI). The 3 most frequently searched terms were “uveitis eye,” “anterior uveitis,” and “uveitis symptoms.” Among the chatbots evaluated, GPT-4 demonstrated the lowest ARI and SMOG scores ( P = .001). Regarding the PEMAT-P, Mistral scored the lowest in understandability, while Grok achieved the highest score for actionability ( P < .001). All chatbots, except Mistral, exhibited high intelligibility scores. GPT-4 had the lowest SMOG and ARI score among the chatbots evaluated, making it the easiest to read. Chatbot technology holds significant potential to enhance healthcare information dissemination and facilitate better patient understanding. While chatbots can effectively provide information on health topics such as uveitis, further improvement is needed to maximize their efficacy and accessibility.Keywords
This publication has 32 references indexed in Scilit:
- Comparison of ChatGPT-4o, Google Gemini 1.5 Pro, Microsoft Copilot Pro, and Ophthalmologists in the management of uveitis and ocular inflammation: A comparative study of large language modelsJournal Français d'Ophtalmologie, 2025
- Responses of Five Different Artificial Intelligence Chatbots to the Top Searched Queries About Erectile Dysfunction: A Comparative AnalysisJournal of Medical Systems, 2024
- Large language models and bariatric surgery patient education: a comparative readability analysis of GPT-3.5, GPT-4, Bard, and online institutional resourcesSurgical Endoscopy, 2024
- A health literacy analysis of online patient-directed educational materials about mycobacterium avium complexJournal of Clinical Tuberculosis and Other Mycobacterial Diseases, 2024
- Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About CancerJAMA Oncology, 2023
- The Use of Large Language Models to Generate Education Materials about UveitisOphthalmology Retina, 2023
- Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google BardEBioMedicine, 2023
- Artificial intelligence-based ChatGPT chatbot responses for patient and parent questions on vernal keratoconjunctivitisGraefe's Archive for Clinical and Experimental Ophthalmology, 2023
- Quality and readability of online information on myofascial pain syndromeJournal of Bodywork and Movement Therapies, 2021
- Simple contents and good readability: Improving health literacy for LEP populationsInternational Journal of Medical Informatics, 2020