Performance of Generative Pretrained Transformer on the National Medical Licensing Examination in Japan
Open Access
- 23 January 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS Digital Health
- Vol. 3 (1) , e0000433
Abstract
The remarkable performance of ChatGPT, launched in November 2022, has significantly impacted the field of natural language processing, inspiring the application of large language models as supportive tools in clinical practice and research worldwide. Although GPT-3.5 recently scored high on the United States Medical Licensing Examination, its performance on medical licensing examinations of other nations, especially non-English speaking nations, has not been sufficiently evaluated. This study assessed GPT’s performance on the National Medical Licensing Examination (NMLE) in Japan and compared it with the actual minimal passing rate for this exam. In particular, the performances of both the GPT-3.5 and GPT-4 models were considered for the comparative analysis. We initially used the GPT models and several prompts for 290 questions without image data from the 116th NMLE (held in February 2022 in Japan) to maximize the performance for delivering correct answers and explanations of the questions. Thereafter, we tested the performance of the best GPT model (GPT-4) with optimized prompts on a dataset of 262 questions without images from the latest 117th NMLE (held in February 2023). The best model with the optimized prompts scored 82.7% for the essential questions and 77.2% for the basic and clinical questions, both of which sufficed the minimum passing scoring rates of 80.0% and 74.6%, respectively. After an exploratory analysis of 56 incorrect answers from the model, we identified the three major factors contributing to the generation of the incorrect answers—insufficient medical knowledge, information on Japan-specific medical system and guidelines, and mathematical errors. In conclusion, GPT-4 with our optimized prompts achieved a minimum passing scoring rate in the latest 117th NMLE in Japan. Beyond its original design of answering examination questions for humans, these artificial intelligence (AI) models can serve as one of the best “sidekicks” for solving problems and addressing the unmet needs in the medical and healthcare fields.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Foundation models for generalist medical artificial intelligenceNature, 2023
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for MedicineNew England Journal of Medicine, 2023
- Artificial Intelligence and Machine Learning in Clinical Medicine, 2023New England Journal of Medicine, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive studyJournal of Educational Evaluation for Health Professions, 2023
- Natural language processing: state of the art, current trends and challengesMultimedia Tools and Applications, 2022
- Natural Language Processing: from Bedside to EverywhereYearbook of Medical Informatics, 2022
- Artificial Intelligence in Current Diabetes Management and PredictionCurrent Diabetes Reports, 2021
- The Power of Scale for Parameter-Efficient Prompt Tuning2021