How does artificial intelligence master urological board examinations? A comparative analysis of different Large Language Models’ accuracy and reliability in the 2022 In-Service Assessment of the European Board of Urology
Authors:
- 10 January 2024
- journal article
- research article
- Published by Springer Nature in World Journal of Urology
- Vol. 42 (1) , 1-10
Abstract
Purpose This study is a comparative analysis of three Large Language Models (LLMs) evaluating their rate of correct answers (RoCA) and the reliability of generated answers on a set of urological knowledge-based questions spanning different levels of complexity. Methods ChatGPT-3.5, ChatGPT-4, and Bing AI underwent two testing rounds, with a 48-h gap in between, using the 100 multiple-choice questions from the 2022 European Board of Urology (EBU) In-Service Assessment (ISA). For conflicting responses, an additional consensus round was conducted to establish conclusive answers. RoCA was compared across various question complexities. Ten weeks after the consensus round, a subsequent testing round was conducted to assess potential knowledge gain and improvement in RoCA, respectively. Results Over three testing rounds, ChatGPT-3.5 achieved RoCa scores of 58%, 62%, and 59%. In contrast, ChatGPT-4 achieved RoCA scores of 63%, 77%, and 77%, while Bing AI yielded scores of 81%, 73%, and 77%, respectively. Agreement rates between rounds 1 and 2 were 84% (κ = 0.67, p < 0.001) for ChatGPT-3.5, 74% (κ = 0.40, p < 0.001) for ChatGPT-4, and 76% (κ = 0.33, p < 0.001) for BING AI. In the consensus round, ChatGPT-4 and Bing AI significantly outperformed ChatGPT-3.5 (77% and 77% vs. 59%, both p = 0.010). All LLMs demonstrated decreasing RoCA scores with increasing question complexity (p < 0.001). In the fourth round, no significant improvement in RoCA was observed across all three LLMs. Conclusions The performance of the tested LLMs in addressing urological specialist inquiries warrants further refinement. Moreover, the deficiency in response reliability contributes to existing challenges related to their current utility for educational purposes.Keywords
This publication has 39 references indexed in Scilit:
- ChatGPT passes German state examination in medicine with picture questions omittedDeutsches Ärzteblatt international, 2023
- ChatGPT Performance on the American Urological Association Self-assessment Study Program and the Potential Influence of Artificial Intelligence in Urologic TrainingUrology, 2023
- Evaluating the Performance of ChatGPT in OphthalmologyOphthalmology Science, 2023
- ChatGPT takes on the European Exam in Core Cardiology: an artificial intelligence success story?European Heart Journal - Digital Health, 2023
- Trialling a Large Language Model (ChatGPT) in General Practice With the Applied Knowledge Test: Observational Study Demonstrating Opportunities and Limitations in Primary CareJMIR Medical Education, 2023
- The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for PapersJMIR Medical Education, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language modelsAnnals of Surgical Treatment and Research, 2023
- ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scopeInternet of Things and Cyber-Physical Systems, 2023