Performance and usability of retrieval-augmented large language models for stroke patient and caregiver support
Open Access
- 1 February 2026
- journal article
- Published by SAGE Publications in Digital Health
- Vol. 12
Abstract
Aim: To develop a Retrieval-Augmented Generation (RAG) question-answering system for stroke patients and family caregivers, and evaluate its performance and usability. Methods: We constructed a localized knowledge base using clinical practice guidelines, consensus, expert opinion, textbooks, systematic review, evidence summary, and peer-reviewed literature. Three LLMs (GPT-4o, Claude3.7, and Qwen3) were first evaluated for accuracy using 184 exam questions under zero-shot and RAG configurations. Thirty open-ended stroke-related questions were assessed by three experienced clinicians across four dimensions. Usability testing was conducted with 20 stroke survivors and family caregivers using the best-performing model, measuring System Usability Scale (SUS) and Net Promoter Score (NPS). Results: RAG integration improved accuracy, relevance, and completeness across all three LLMs, with GPT-4o under RAG configuration achieving the highest overall mean score. However, the addition of RAG slightly reduced understandability for Claude3.7 and Qwen3. Usability testing yielded high acceptance. Conclusions: RAG can enhance the reliability of LLM-generated responses in stroke-related questions, offering trusted, guideline-based information. The high usability ratings suggest early feasibility for real-world deployment, while future research should assess its linguistic accessibility and long-term clinical benefits in real-world caregiving contexts.Keywords
Funding Information
- National Natural Science Foundation of China (72304071)
- Nursing Research Fund of the First Affiliated Hospital of Ningbo University (H2025YJ002)
- Medical Science and Technology Project of Zhejiang Province (2025HY0957)
This publication has 33 references indexed in Scilit:
- Current Status of ChatGPT Use in Medical Education: Potentials, Challenges, and StrategiesJournal of Medical Internet Research, 2024
- Detecting hallucinations in large language models using semantic entropyNature, 2024
- Assessing the research landscape and clinical utility of large language models: a scoping reviewBMC Medical Informatics and Decision Making, 2024
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023
- The ultimate question? Evaluating the use of Net Promoter Score in healthcare: A systematic reviewHealth Expectations, 2022
- Temporal trend and attributable risk factors of stroke burden in China, 1990–2019: an analysis for the Global Burden of Disease Study 2019The Lancet Public Health, 2021
- Artificial Intelligence-Based Conversational Agents for Chronic Conditions: Systematic Literature ReviewJournal of Medical Internet Research, 2020
- Comparison of confidence interval methods for an intra-class correlation coefficient (ICC)BMC Medical Research Methodology, 2014
- Evidence-Based Community Stroke RehabilitationStroke, 2013