Generating multiple-choice questions using reverse engineering techniques
Open Access
- 9 January 2026
- journal article
- Published by Taylor & Francis in Medical Education Online
- Vol. 31 (1) , 2611632
Abstract
The study is aimed at exploring the potential of using ChatGPT-4 to develop multiple-choice questions that adhere to item development principles through the application of reverse engineering techniques in nursing education. To determine whether ChatGPT-4 could generate multiple-choice questions regarding medical-surgical nursing, the researchers first evaluated ChatGPT-4's level of nursing knowledge. The researchers then developed 20 prompts using reverse engineering techniques; these were subsequently entered into ChatGPT-4, resulting in the generation of 60 multiple-choice questions. All 60 questions were reviewed by five nursing education experts to evaluate compliance with item development principles. ChatGPT-4 demonstrated a high accuracy rate (98.00%) when answering multiple-choice questions about medical-surgical nursing education. AI-generated questions primarily adhered to the principles of item development, with an average compliance score of 7.90/9.00. The least adhered to among the nine-question development principles was 'The question stem should use only essential words or phrases.' Given that the AI-generated questions may serve as preliminary drafts for use in nursing education following expert review, ChatGPT-4 can function as a valuable tool in the design of educational materials. When utilizing ChatGPT-4, it is beneficial for nursing educators to apply reverse engineering techniques that enhance question quality through repeated comparison, as well as to analyze the prompts and corresponding AI-generated questions. Furthermore, including structured outlines and representative sample multiple-choice questions using Markdown syntax in the prompt can enhance the quality of AI-generated questions.Keywords
Funding Information
- National Research Foundation of Korea (2023R1A2C2006838)
This publication has 29 references indexed in Scilit:
- Prompt engineering when using generative AI in nursing educationNurse Education in Practice, 2023
- Performance of ChatGPT on Registered Nurse License Exam in Taiwan: A Descriptive StudyHealthcare, 2023
- Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4BMC Medical Education, 2023
- Revolutionizing nursing education through Ai integration: A reflection on the disruptive impact of ChatGPTNurse Education Today, 2023
- Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation StudyJMIR Nursing, 2023
- ChatGPT can pass the AHA exams: Open-ended questions outperform multiple-choice formatResuscitation, 2023
- Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A PrimerFrontiers in Public Health, 2018
- Pushing Critical Thinking Skills With Multiple-Choice Questions: Does Bloom’s Taxonomy Work?Academic Medicine, 2018
- A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samplesBMC Medical Research Methodology, 2013
- Interrater reliability: the kappa statisticBiochemia Medica, 2012