Large Language Models Evaluation of Medical Licensing Examination Using GPT-4.0, ERNIE Bot 4.0, and GPT-4o
Top Cited Papers
Open Access
- 17 January 2026
- journal article
- research article
- Published by MDPI AG in Bioengineering
- Vol. 13 (1) , 113
Abstract
This study systematically evaluated the performance of three advanced large language models (LLMs)—GPT-4.0, ERNIE Bot 4.0, and GPT-4o—in the 2023 Chinese Medical Licensing Examination. Employing a dataset of 600 standardized questions, we analyzed the accuracy of each model in answering questions from three comprehensive sections: Basic Medical Comprehensive, Clinical Medical Comprehensive, and Humanities and Preventive Medicine Comprehensive. Our results demonstrate that both ERNIE Bot 4.0 and GPT-4o significantly outperformed GPT-4.0, achieving accuracies above the national pass mark. The study further examined the strengths and limitations of each model, providing insights into their applicability in medical education and potential areas for future improvement. These findings underscore the promise and challenges of deploying LLMs in multilingual medical education, suggesting a pathway towards integrating AI into medical training and assessment practices.Keywords
Funding Information
- Science and Technology Program of the Health Commission of Fujian Province, China (2022QN01010219)
This publication has 44 references indexed in Scilit:
- Comparative Assessment of Otolaryngology Knowledge Among Large Language ModelsThe Laryngoscope, 2024
- OpenAI’s GPT-4o in surgical oncology: Revolutionary advances in generative artificial intelligenceEuropean Journal of Cancer, 2024
- The application of large language models in medicine: A scoping reviewiScience, 2024
- Accuracy and Reliability of Chatbot Responses to Physician QuestionsJAMA Network Open, 2023
- Effects of a comprehensive brain computed tomography deep learning model on radiologist detection accuracyEuropean Radiology, 2023
- ChatGPT Performs on the Chinese National Medical Licensing ExaminationJournal of Medical Systems, 2023
- How large language models including generative pre-trained transformer (GPT) 3 and 4 will impact medicine and surgeryTechniques in Coloproctology, 2023
- Large language models encode clinical knowledgeNature, 2023
- ChatGPT: not all languages are equalNature, 2023
- Classification score approach for detecting adversarial example in deep neural networkMultimedia Tools and Applications, 2020