Analysis of Responses of GPT-4 V to the Japanese National Clinical Engineer Licensing Examination
- 11 September 2024
- journal article
- research article
- Published by Springer Nature in Journal of Medical Systems
- Vol. 48 (1) , 1-9
Abstract
Chat Generative Pretrained Transformer (ChatGPT; OpenAI) is a state-of-the-art large language model that can simulate human-like conversations based on user input. We evaluated the performance of GPT-4 V in the Japanese National Clinical Engineer Licensing Examination using 2,155 questions from 2012 to 2023. The average correct answer rate for all questions was 86.0%. In particular, clinical medicine, basic medicine, medical materials, biological properties, and mechanical engineering achieved a correct response rate of ≥ 90%. Conversely, medical device safety management, electrical and electronic engineering, and extracorporeal circulation obtained low correct answer rates ranging from 64.8% to 76.5%. The correct answer rates for questions that included figures/tables, required numerical calculation, figure/table ∩ calculation, and knowledge of Japanese Industrial Standards were 55.2%, 85.8%, 64.2% and 31.0%, respectively. The reason for the low correct answer rates is that ChatGPT lacked recognition of the images and knowledge of standards and laws. This study concludes that careful attention is required when using ChatGPT because several of its explanations lack the correct description.Keywords
This publication has 28 references indexed in Scilit:
- Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of CanadaFamily Medicine and Community Health, 2024
- Performance of ChatGPT on the National Korean Occupational Therapy Licensing ExaminationDigital Health, 2024
- Performance of ChatGPT on Stage 1 of the Taiwanese medical licensing examDigital Health, 2024
- ChatGPT in Iranian medical licensing examination: evaluating the diagnostic accuracy and decision-making capabilities of an AI-based modelBMJ Health & Care Informatics, 2023
- Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final ExaminationScientific Reports, 2023
- ChatGPT Performs on the Chinese National Medical Licensing ExaminationJournal of Medical Systems, 2023
- Performance of ChatGPT on the pharmacist licensing examination in TaiwanJournal of the Chinese Medical Association, 2023
- ChatGPT and Other Large Language Models Are Double-edged SwordsRadiology, 2023
- Using ChatGPT to write patient clinic lettersThe Lancet Digital Health, 2023
- Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer informationJNCI Cancer Spectrum, 2023