Evaluating the Accuracy of Medical Information Generated by ChatGPT and Gemini and Its Alignment With International Clinical Guidelines From the Surviving Sepsis Campaign: Comparative Study
Open Access
- 17 December 2025
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 9 (1) , e84251
Abstract
Background: Assessment of medical information provided by artificial intelligence (AI) chatbots like ChatGPT and Google’s Gemini and comparison with international guidelines is a burgeoning area of research. These AI models are increasingly being considered for their potential to support clinical decision-making and patient education. However, their accuracy and reliability in delivering medical information that aligns with established guidelines remain under scrutiny. Objective: This study aims to assess the accuracy of medical information generated by ChatGPT and Gemini and its alignment with international guidelines for sepsis management. Methods: ChatGPT and Gemini were asked 18 questions about the Surviving Sepsis Campaign guidelines, and the responses were evaluated by 7 independent intensive care physicians. The responses generated were scored as follows: 3=correct, complete, and accurate; 2=correct but incomplete or inaccurate; and 1=incorrect. This scoring system was chosen to provide a clear and straightforward assessment of the accuracy and completeness of the responses. The Fleiss κ test was used to assess the agreement between evaluators, and the Mann-Whitney U test was used to test for the significance of differences between the correct responses generated by ChatGPT and Gemini. Results: ChatGPT provided 5 (28%) perfect responses, 12 (67%) nearly perfect responses, and 1 (5%) low-quality response, with substantial agreement among the evaluators (Fleiss κ=0.656). Gemini, on the other hand, provided 3 (17%) perfect responses, 14 (78%) nearly perfect responses, and 1 (5%) low-quality response, with moderate agreement among the evaluators (Fleiss κ=0.582). The Mann-Whitney U test revealed no statistically significant difference between the two platforms (P=.48). Conclusions: ChatGPT and Gemini both demonstrated potential for generating medical information. Despite their current limitations, both showed promise as complementary tools in patient education and clinical decision-making. The medical information generated by ChatGPT and Gemini still needs ongoing evaluation regarding its accuracy and alignment with international guidelines in different medical domains, particularly in the sepsis field.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Can artificial intelligence models serve as patient information consultants in orthodontics?BMC Medical Informatics and Decision Making, 2024
- An evaluation of ChatGPT and Bard (Gemini) in the context of biological knowledge retrievalAccess Microbiology, 2024
- Use of artificial intelligence chatbots in clinical management of immune-related adverse eventsJournal for ImmunoTherapy of Cancer, 2024
- Evaluation of ChatGPT in providing appropriate fracture prevention recommendations and medical science question responses: A quantitative researchMedicine, 2024
- Google DeepMind’s gemini AI versus ChatGPT: a comparative analysis in ophthalmologyEye, 2024
- Assessment of Artificial Intelligence Chatbot Responses to Top Searched Queries About CancerJAMA Oncology, 2023
- Use of Artificial Intelligence Chatbots for Cancer Treatment InformationJAMA Oncology, 2023
- Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research ScenariosJournal of Medical Systems, 2023
- ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scopeInternet of Things and Cyber-Physical Systems, 2023
- Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock 2021Critical Care Medicine, 2021