ChatGPT as a tool for reviewing multiple-choice questions in the health sector
Authors:
Open Access
- 13 May 2026
- journal article
- Published by Springer Nature in Scientific Reports
- Vol. 16 (1)
Abstract
Artificial Intelligence (AI), particularly ChatGPT-4, offers promising applications in medical education, including multiple-choice question (MCQ) development. This study aimed to evaluate and compare the quality of 36 MCQs created by medical faculty with their versions reviewed by ChatGPT-4. A cross-sectional, quantitative approach was used. Ten external health education specialists and four study authors (internal evaluators) assessed the questions based on 38 criteria. While external evaluators found no statistically significant difference in criteria met between versions (p = 0.325), the study authors, who underwent standardization meetings, identified a statistically significant increase in the number of criteria met by ChatGPT-4-reviewed MCQs (p < 0.001). Descriptive statistics, Wilcoxon Signed-Rank Test, and Non-Metric Multidimensional Scaling were employed. The results showed that ChatGPT-4 demonstrated proficiency in modifying questions to reflect greater structural clarity and adherence to basic item-writing principles, resulting in questions with increased clarity and objectivity. However, it struggled to incorporate clinical reasoning and higher-order thinking when these were lacking, particularly given the non-optimized prompt used. Despite these limitations, AI's revisions were aligned with faculty quality standards, demonstrating its potential to complement faculty efforts, emphasizing the critical role of calibrated human expertise and effective prompt engineering, rather than replacement.Keywords
This publication has 29 references indexed in Scilit:
- The pitfalls of multiple-choice questions in generative AI and medical educationScientific Reports, 2025
- ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature reviewPostgraduate Medical Journal, 2024
- A scoping review of artificial intelligence in medical education: BEME Guide No. 84Medical Teacher, 2024
- ChatGPT and Generative Artificial Intelligence for Medical Education: Potential Impact and OpportunityAcademic Medicine, 2023
- The next paradigm shift? ChatGPT, artificial intelligence, and medical educationMedical Teacher, 2023
- Learning to Reuse Distractors to Support Multiple-Choice Question Generation in EducationIEEE Transactions on Learning Technologies, 2022
- Supervision and feedback for junior medical staff in Australian emergency departments: findings from the emergency medicine capacity assessment studyBMC Medical Education, 2010
- The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studiesThe Lancet, 2007
- The Effects of Violating Standard Item Writing Principles on Tests and Students: The Consequences of Using Flawed Test Items on Achievement Examinations in Medical EducationAdvances in Health Sciences Education, 2005
- Cross institutional collaboration in assessment: a case on progress testingMedical Teacher, 2004