Evaluating the value of AI-generated questions for USMLE step 1 preparation: A study using ChatGPT-3.5
Authors:
- 27 March 2025
- journal article
- Published by Taylor & Francis in Medical Teacher
- Vol. 47 (10) , 1645-1653
Abstract
Students are increasingly relying on artificial intelligence (AI) for medical education and exam preparation. However, the factual accuracy and content distribution of AI-generated exam questions for self-assessment have not been systematically investigated. Curated prompts were created to generate multiple-choice questions matching the USMLE Step 1 examination style. We utilized ChatGPT-3.5 to generate 50 questions and answers based upon each prompt style. We manually examined output for factual accuracy, Bloom's Taxonomy, and category within the USMLE Step 1 content outline. ChatGPT-3.5 generated 150 multiple-choice case-style questions and selected an answer. Overall, 83% of generated multiple questions had no factual inaccuracies and 15% contained one to two factual inaccuracies. With simple prompting, common themes included deep venous thrombosis, myocardial infarction, and thyroid disease. Topic diversity improved by separating content topic generation from question generation, and specificity to Step 1 increased by indicating that "treatment" questions were not desired. We demonstrate that ChatGPT-3.5 can successfully generate Step 1 style questions with reasonable factual accuracy, and this method may be used by medical students preparing for USMLE examinations. While AI-generated questions demonstrated adequate factual accuracy, targeted prompting techniques should be used to overcome ChatGPT's bias towards particular medical conditions.Keywords
This publication has 31 references indexed in Scilit:
- Evaluation of a Generative Language Model Tool for Writing Examination QuestionsAmerican Journal of Pharmaceutical Education, 2024
- Chat-GPT on brain tumors: An examination of Artificial Intelligence/Machine Learning’s ability to provide diagnoses and treatment plans for example neuro-oncology casesClinical Neurology and Neurosurgery, 2024
- ChatGPT 3.5 fails to write appropriate multiple choice practice exam questionsAcademic Pathology, 2023
- Preclerkship Medical Students’ Use of Third-Party Learning ResourcesJAMA Network Open, 2023
- The Current and Future State of AI Interpretation of Medical ImagesNew England Journal of Medicine, 2023
- Passing is Great: Can ChatGPT Conduct USMLE Exams?Annals of Biomedical Engineering, 2023
- Large language models (LLM) and ChatGPT: what will the impact on nuclear medicine be?European Journal of Nuclear Medicine and Molecular Imaging, 2023
- A Chat(GPT) about the future of scientific publishingBrain, Behavior, and Immunity, 2023
- Applying Bloom's taxonomy in clinical surgery: Practical examplesAnnals of Medicine & Surgery, 2021
- An Overview of Chatbot Technology2020