Large language model chatbots for patient education in kidney stones: a scoping review
- 29 October 2025
- journal article
- review article
- Published by Springer Nature in World Journal of Urology
- Vol. 43 (1) , 1-20
Abstract
Purpose In 2024, 17% of adults reported using an artificial intelligence (AI) chatbot at least once a month as a source of health information, rising to 25% among those under 30. We aim to conduct a scoping review of the existing literature assessing the performance of large language model (LLM) chatbots for patient education in kidney stone disease (KSD). Methods The Joanna Briggs Institute methodology was followed. Ovid MEDLINE, Embase, CENTRAL, Web of Science, CINAHL, and Google Scholar were searched for studies in all languages, from 2015 up to February 16th 2025, evaluating LLM chatbots to create educational content on KSD. Two independent reviewers completed screening, full-text review, and data extraction, with conflicts resolved by a third reviewer. Results Of the 281 search results, 17 were included. Five of six studies assessing readability found that LLM responses exceeded the recommended 6th-8th grade reading level, though effective prompting can help meet this target. Six out of eight studies reported adequate to very good accuracy performance, with two showing comparable performance to traditional information sources. Understandability and actionability performance was poor. Quality performance was variable across seven studies. Three studies assessed patients’ perception, revealing a mixed but generally favorable experience. Two studies noted notable deviations from established clinical guidelines. Conclusion LLM chatbots show potential for KSD patient education and physician workload reduction, but currently have limitations in readability, guideline adherence, understandability, and actionability. This could be mitigated by prompting and the development of urology-specific tools trained on validated content and evaluated with patient involvement.Keywords
This publication has 43 references indexed in Scilit:
- Battle of the bots: a comparative analysis of ChatGPT and bing AI for kidney stone-related questionsWorld Journal of Urology, 2024
- What patients with kidney stones believe about their conditionUrolithiasis, 2024
- Artificial intelligence in urology: a technology with transformative potentialTrends in Urology & Men's Health, 2024
- ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A ReviewCognitive Computation, 2024
- Evaluating the Performance of Different Large Language Models on Health Consultation and Patient Education in UrolithiasisJournal of Medical Systems, 2023
- Evaluating the performance of ChatGPT in answering questions related to urolithiasisInternational Urology and Nephrology, 2023
- ChatGPT and most frequent urological diseases: analysing the quality of information and potential risks for patientsWorld Journal of Urology, 2023
- AI Chatbots, Health Privacy, and Challenges to HIPAA ComplianceJAMA, 2023
- An objective analysis of quality and readability of online information on COVID-19Health and Technology, 2021
- Burden of Urolithiasis: Trends in Prevalence, Treatments, and CostsEuropean Urology Focus, 2017