Performance of Large Language Models in Medical Exams: A Comparison Between ChatGPT and Medical Students
- 22 December 2025
- journal article
- Published by Tip Egitimi Dunyasi in Tıp Eğitimi Dünyası
- Vol. 24 (74) , 135-143
Abstract
Background: Medical education in Türkiye is delivered through a six-year, discipline-based curriculum aligned with global trends. The assessment process largely relies on multiple-choice questions, placing a significant preparation burden on faculty members. AI-powered large language models like ChatGPT have the potential to ease exam preparation, enhance feedback quality, and support personalized learning. The aim of this study is to evaluate the success of the ChatGPT-4o model performs when answering multiple-choice (MCQ) questions on medical education exams. Additionally, by comparing exam performance and consistency to student success, we explore the potential benefits of AI-supported models to medical education. Methods: This cross-sectional, analytical investigation was carried out in Türkiye at the [XX] University Faculty of Medicine. During the 2023–2024 academic year, ChatGPT solved multiple-choice questions from seven board exams and one final exam for third-year students, the results were compared with the students' achievements. Statistical analysis included descriptive statistics, correlation analyses, chi-square tests, McNemar tests, and t-tests for independent samples. Results: With a 90.2% correct response percentage, ChatGPT outperformed the entire class, outperforming 293 other students. There was no significant difference in the correct response rate between the surgical, internal, and fundamental medical sciences (p = 0.742). In several fields, such psychiatry, neurology, and medical genetics, 100% success was attained. Forensic medicine, family medicine, medical ethics, pulmonary medicine, and thoracic surgery all had success rates that were lower than 80%. A retest conducted two months later revealed that ChatGPT's success rate had somewhat risen, with response consistency standing at 91.4%. Conclusions: With a high success rate on medical education tests, ChatGPT has shown a great deal of promise to help both students and instructors. The integration of AI models into educational systems should be done strategically and with a human-centered approach, though, given the constraints in areas like clinical reasoning, ethical evaluation, and human-centered medical education. It is important to design instructional strategies in the future that combine artificial intelligence technologies with human skills.Keywords
This publication has 24 references indexed in Scilit:
- Performance of single-agent and multi-agent language models in Spanish language medical competency examsBMC Medical Education, 2025
- Chat GPT 4o vs residents: French language evaluation in ophthalmologyAJO International, 2025
- Artificial Intelligence and Decision-Making in Healthcare: A Thematic Analysis of a Systematic Review of ReviewsHealth Services Research and Managerial Epidemiology, 2024
- Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final ExaminationScientific Reports, 2023
- Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessmentsScientific Reports, 2023
- ChatGPT in medical school: how successful is AI in progress testing?Medical Education Online, 2023
- Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning DocumentationJournal of General Internal Medicine, 2022
- Evolution and Revolution in Artificial Intelligence in EducationInternational Journal of Artificial Intelligence in Education, 2016
- Joining a conversation: the problem/gap/hook heuristicPerspectives on Medical Education, 2015
- Writing Multiple-Choice Questions for Continuing Medical Education Activities and Self-Assessment ModulesRadioGraphics, 2006