A Comparative Analysis of ChatGPT, ChatGPT-4, and Google Bard Performances at the Advanced Burn Life Support Exam
- 4 June 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in Journal of Burn Care & Research
- Vol. 45 (4) , 945-948
Abstract
Artificial intelligence and large language models (LLMs) have recently gained attention as promising tools in various healthcare domains, offering potential benefits in clinical decision-making, medical education, and research. The Advanced Burn Life Support (ABLS) program is a didactic initiative endorsed by the American Burn Association, aiming to provide knowledge on the immediate care of severely burn patients. The aim of the study was to compare the performance of three LLMs (ChatGPT-3.5, ChatGPT-4, and Google Bard) on the ABLS exam. The ABLS exam consists of 50 questions with 5 multiple-choice answers. The passing threshold is 80% of correct answers. The 3 LLMs were queried with the 50 questions included in the latest version of the ABLS exam, on July 18th, 2023. ChatGPT-3.5 scored 86% (43 out of 50), ChatGPT-4 scored 90% (45 out of 50), and Bard scored 70% (35 out of 50). No difference was measured between ChatGPT-3.5 and ChatGPT-4 (P = .538) and between ChatGPT-3.5 and Bard (P = .054), despite the borderline P-value. ChatGPT-4 performed significantly better than Bard (P = .012). Out of the 50 questions, 78% (n = 39) were direct questions, while 12% (n = 11) were presented as clinical scenarios. No difference in the rate of wrong answers was found based on the type of question for the 3 LLMs. ChatGPT-3.5 and ChatGPT-4 demonstrated high accuracy at the ABLS exam and outperformed Google Bard. However, the potential multiple applications of LLMs in emergency burn and trauma care necessitate appropriate surveillance and most likely should represent a tool to complement human cognition.Keywords
This publication has 4 references indexed in Scilit:
- ChatGPT in orthopedics: a narrative review exploring the potential of artificial intelligence in orthopedic practiceFrontiers in Surgery, 2023
- Revolutionizing patient education: ChatGPT outperforms Google in answering patient queries on free flap reconstructionMicrosurgery, 2023
- The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for PapersJMIR Medical Education, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023