Performance of ChatGPT-3.5 and ChatGPT-4 on the European Board of Urology (EBU) exams: a comparative analysis
Authors:
- 26 July 2024
- journal article
- research article
- Published by Springer Nature in World Journal of Urology
- Vol. 42 (1) , 1-7
Abstract
Background and objective In the transformative era of artificial intelligence, its integration into various spheres, especially healthcare, has been promising. The objective of this study was to analyze the performance of ChatGPT, as open-source Large Language Model (LLM), in its different versions on the recent European Board of Urology (EBU) in-service assessment questions. Design and setting We asked multiple choice questions of the official EBU test books to ChatGPT-3.5 and ChatGPT-4 for the following exams: exam 1 (2017–2018), exam 2 (2019–2020) and exam 3 (2021–2022). Exams were passed with ≥60% correct answers. Results ChatGPT-4 provided significantly more correct answers in all exams compared to the prior version 3.5 (exam 1: ChatGPT-3.5 64.3% vs. ChatGPT-4 81.6%; exam 2: 64.5% vs. 80.5%; exam 3: 56% vs. 77%, p < 0.001, respectively). Test exam 3 was the only exam ChatGPT-3.5 did not pass. Within the different subtopics, there were no significant differences of provided correct answers by ChatGPT-3.5. Concerning ChatGPT-4, the percentage in test exam 3 was significantly decreased in the subtopics Incontinence (exam 1: 81.6% vs. exam 3: 53.6%; p = 0.026) and Transplantation (exam 1: 77.8% vs. exam 3: 0%; p = 0.020). Conclusion Our findings indicate that ChatGPT, especially ChatGPT-4, has the general ability to answer complex medical questions and might pass FEBU exams. Nevertheless, there is still the indispensable need for human validation of LLM answers, especially concerning health care issues.Keywords
This publication has 20 references indexed in Scilit:
- How does artificial intelligence master urological board examinations? A comparative analysis of different Large Language Models’ accuracy and reliability in the 2022 In-Service Assessment of the European Board of UrologyWorld Journal of Urology, 2024
- Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical trainingScientific Reports, 2023
- Beyond human in neurosurgical exams: ChatGPT's success in the Turkish neurosurgical society proficiency board examsComputers in Biology and Medicine, 2023
- Development of a Personalized Chat Model Based on the European Association of Urology Oncology Guidelines: Harnessing the Power of Generative Artificial Intelligence in Clinical PracticeEuropean Urology Oncology, 2023
- Large language models encode clinical knowledgeNature, 2023
- The imperative for regulatory oversight of large language models (or generative AI) in healthcarenpj Digital Medicine, 2023
- Performance of Generative Large Language Models on Ophthalmology Board–Style QuestionsAmerican Journal of Ophthalmology, 2023
- ChatGPT failed Taiwan’s Family Medicine Board ExamJournal of the Chinese Medical Association, 2023
- Is ChatGPT an Evidence-based Doctor?European Urology, 2023
- Artificial intelligence in urological oncology: An update and future applicationsUrologic Oncology: Seminars and Original Investigations, 2021