Can Large Language Models Support Family Communication in the Intensive Care Unit? A Comparative Accuracy Study
Open Access
- 30 June 2026
- journal article
- Published by Sakarya Tip Dergisi in Sakarya Medical Journal
- Vol. 16 (2) , 200-207
Abstract
Objective: Family members of patients admitted to the intensive care unit (ICU) frequently experience uncertainty, emotional distress, and significant informational needs. Communication gaps remain common in ICU practice due to time constraints and clinical complexity. Large language models (LLMs) may offer scalable support for patient–family information delivery; however, their performance in responding to real-world ICU family questions has not been systematically evaluated.Methods: This evaluator-blinded, cross-sectional study compared the accuracy of responses generated by five widely used LLMs (Claude Sonnet 4.0, ChatGPT 5.0, Gemini 2.5, Grok-4, and Sonar) to questions commonly asked by ICU family members. A standardized set of 25 questions was generated by prompting each model to list frequently asked ICU family questions. All questions were subsequently posed to all five models in blinded, independent sessions. Two intensive care medicine specialists independently rated response accuracy using a 6-point Likert scale. Inter-rater reliability was assessed using Cohen’s kappa. Differences between models were analyzed using the Friedman test with post-hoc Wilcoxon signed-rank tests.Results: A total of 125 responses were evaluated. Inter-rater agreement was moderate (Cohen’s κ = 0.56; overall agreement 73.6%). Accuracy scores differed significantly among models (p < 0.001). Claude Sonnet 4.0 achieved the highest mean accuracy score (5.66 ± 0.61), followed by ChatGPT 5.0, Gemini 2.5, and Sonar, with no statistically significant differences among these four models. Grok-4 demonstrated significantly lower accuracy compared with all other models (all p < 0.001).Conclusions: Most contemporary LLMs demonstrated high accuracy in answering questions commonly posed by ICU family members, although performance varied across platforms. Selected LLMs may serve as supportive tools to reinforce clinician–family communication; however, careful model selection, clinical oversight, and ethical safeguards are required before implementation in high-stakes intensive care settings.Keywords
This publication has 24 references indexed in Scilit:
- Comparative analysis of large language models in providing patient information about keratoconus and contact lensesInternational Ophthalmology, 2025
- Large Language Models for Chatbot Health Advice StudiesJAMA Network Open, 2025
- Large language models in critical careJournal of Intensive Medicine, 2024
- A review of Explainable Artificial Intelligence in healthcareComputers and Electrical Engineering, 2024
- Ethical and regulatory challenges of large language models in medicineThe Lancet Digital Health, 2024
- Large language models encode clinical knowledgeNature, 2023
- Assessing the usefulness of a large language model to query and summarize unstructured medical notes in intensive careIntensive Care Medicine, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- The ethics of AI in health care: A mapping reviewSocial Science & Medicine, 2020
- High-performance medicine: the convergence of human and artificial intelligenceNature Medicine, 2019