AI chatbots show promise but limitations on UK medical exam questions: a comparative performance study
Open Access
- 14 August 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-11
Abstract
Large language models (LLMs) like ChatGPT have potential applications in medical education such as helping students study for their licensing exams by discussing unclear questions with them. However, they require evaluation on these complex tasks. The purpose of this study was to evaluate how well publicly accessible LLMs performed on simulated UK medical board exam questions. 423 board-style questions from 9 UK exams (MRCS, MRCP, etc.) were answered by seven LLMs (ChatGPT-3.5, ChatGPT-4, Bard, Perplexity, Claude, Bing, Claude Instant). There were 406 multiple-choice, 13 true/false, and 4 "choose N" questions covering topics in surgery, pediatrics, and other disciplines. The accuracy of the output was graded. Statistics were used to analyze differences among LLMs. Leaked questions were excluded from the primary analysis. ChatGPT 4.0 scored (78.2%), Bing (67.2%), Claude (64.4%), and Claude Instant (62.9%). Perplexity scored the lowest (56.1%). Scores differed significantly between LLMs overall (p < 0.001) and in pairwise comparisons. All LLMs scored higher on multiple-choice vs true/false or “choose N” questions. LLMs demonstrated limitations in answering certain questions, indicating refinements needed before primary reliance in medical education. However, their expanding capabilities suggest a potential to improve training if thoughtfully implemented. Further research should explore specialty specific LLMs and optimal integration into medical curricula.Keywords
Funding Information
- Misr University for Science & Technology
This publication has 19 references indexed in Scilit:
- Data leakage in deep learning studies of translational EEGFrontiers in Neuroscience, 2024
- Performance of ChatGPT on Specialty Certificate Examination in Dermatology multiple-choice questionsClinical and Experimental Dermatology, 2023
- Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and LimitationsRadiology, 2023
- Large Language Models in Medical Education: Opportunities, Challenges, and Future DirectionsJMIR Medical Education, 2023
- Evaluating the Performance of ChatGPT in OphthalmologyOphthalmology Science, 2023
- Performance of ChatGPT on UK Standardized Admission Tests: Insights From the BMAT, TMUA, LNAT, and TSA ExaminationsJMIR Medical Education, 2023
- ChatGPT passing USMLE shines a spotlight on the flaws of medical educationPLOS Digital Health, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- Inflation of test accuracy due to data leakage in deep learning-based classification of OCT imagesScientific Data, 2022
- Artificial intelligence in medicineThe Annals of The Royal College of Surgeons of England, 2004