Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study
Top Cited Papers
Open Access
- 8 February 2024
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 10 (1) , e50965
Abstract
Background: The potential of artificial intelligence (AI)–based large language models, such as ChatGPT, has gained significant attention in the medical field. This enthusiasm is driven not only by recent breakthroughs and improved accessibility, but also by the prospect of democratizing medical knowledge and promoting equitable health care. However, the performance of ChatGPT is substantially influenced by the input language, and given the growing public trust in this AI tool compared to that in traditional sources of information, investigating its medical accuracy across different languages is of particular importance. Objective: This study aimed to compare the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination. Methods: To assess GPT-3.5’s and GPT-4's medical proficiency, we used 937 original multiple-choice questions from 3 written German medical licensing examinations in October 2021, April 2022, and October 2022. Results: GPT-4 achieved an average score of 85% and ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the same examinations in October 2021, April 2022, and October 2022, respectively. This represents a substantial improvement of 27% compared to GPT-3.5, which only passed 1 out of the 3 examinations. While GPT-3.5 performed well in psychiatry questions, GPT-4 exhibited strengths in internal medicine and surgery but showed weakness in academic research. Conclusions: The study results highlight ChatGPT’s remarkable improvement from moderate (GPT-3.5) to high competency (GPT-4) in answering medical licensing examination questions in German. While GPT-4’s predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results. As the replacement of search engines by AI tools seems possible in the future, further studies with nonprofessional questions are needed to assess the safety and accuracy of ChatGPT for the general population.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019The Lancet Psychiatry, 2022
- What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical ExamsApplied Sciences, 2021
- Reproducible Summary Tables with the gtsummary PackageThe R Journal, 2021
- Welcome to the TidyverseJournal of Open Source Software, 2019
- Risk prediction using natural language processing of electronic mental health records in an inpatient forensic psychiatry settingJournal of Biomedical Informatics, 2018
- Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materialsJournal of the Medical Library Association, 2018
- Kompetenzorientiert prüfen im Staatsexamen MedizinBundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz, 2017
- ggplot22016
- Using electronic medical records to enable large-scale studies in psychiatry: treatment resistant depression as a modelPsychological Medicine, 2011
- pROC: an open-source package for R and S+ to analyze and compare ROC curvesBMC Bioinformatics, 2011