Psychometric Evaluation of Automatically Generated Template-Based Psychiatry Questions for Medical Students: A Validity and Reliability Study
- 22 December 2025
- journal article
- Published by Tip Egitimi Dunyasi in Tıp Eğitimi Dünyası
- Vol. 24 (74) , 209-216
Abstract
Background: Multiple-choice questions (MCQs) are widely used in medical education due to their objectivity, efficiency, and ability to cover a broad knowledge base. Case-based MCQs provide additional benefits by evaluating students’ clinical reasoning and decision-making skills. In psychiatry education, unique challenges arise from overlapping symptoms, reliance on subjective reports, and the absence of objective diagnostic tools. The aim of this study was to administer MCQs generated through template-based automatic item generation (AIG) in psychiatry to medical students and to evaluate their psychometric properties (difficulty and discrimination indices). Methods: Following ethical approval from XXX University Ethics Committee, the study included 138 volunteer students (61.6%) from a total of 224 who completed psychiatry clerkship during the 2023–2024 and 2024–2025 academic years. From a pool of 1189 template-based automatically generated questions, 22 were randomly selected to form the exam. The test was administered face-to-face under supervision, and students were not informed of the origin of the items. Difficulty indices were calculated as the proportion of correct answers, while discrimination indices were computed by comparing the performance of the top 27% and bottom 27% groups. Results: The mean exam score was 15.21 ± 3.55 out of 22. The average difficulty index was 0.69, classifying the exam as “easy.” Of the items, 63.6% were very easy, 9.1% easy, and 27.3% moderate. The most difficult item concerned somatization (0.33), whereas the easiest was related to bipolar disorder (0.92). Discrimination indices ranged from 0.19 to 0.70, with an average of 0.37. Ten items (45.6%) demonstrated excellent discrimination, eleven (50%) acceptable, and one (4.5%) poor. The highest discrimination was observed in the schizophreniform disorder item (0.70), while the lowest was in the postpartum psychosis item (0.19). Conclusions: This study represents the first direct implementation of template-based AIG in Turkish psychiatry education. The findings demonstrated that automatically generated MCQs achieved acceptable psychometric standards in terms of both difficulty and discrimination. Template-based AIG may reduce faculty workload while ensuring consistent and high-quality question development. However, further refinement is needed to generate items assessing higher-order cognitive processes. Multicenter comparative studies could provide stronger evidence for the integration of AIG into medical education assessments.Keywords
Funding Information
- No.
This publication has 17 references indexed in Scilit:
- ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: a literature reviewPostgraduate Medical Journal, 2024
- Feasibility assurance: a review of automatic item generation in medical assessmentAdvances in Health Sciences Education, 2022
- Clinical Reasoning Assessment Methods: A Scoping Review and Practical GuidanceAcademic Medicine, 2019
- Non-technical skills assessments in undergraduate medical education: A focused BEME systematic review: BEME Guide No. 54Medical Teacher, 2019
- Plus ça change, plus c’est pareil: Making a continued case for the use of MCQs in medical educationMedical Teacher, 2018
- Pushing Critical Thinking Skills With Multiple-Choice Questions: Does Bloom’s Taxonomy Work?Academic Medicine, 2018
- Feature Selection Using Multi-objective Optimization for Aspect Based Sentiment Analysis2017
- An Educational Model for Undergraduate Psychiatry Students to Promote Clinical Diagnostic ReasoningProcedia - Social and Behavioral Sciences, 2014
- Logic Structure of Clinical Judgment and Its Relation to Medical and Psychiatric SemiologyPsychopathology, 2012
- Using automatic item generation to create multiple‐choice test itemsMedical Education, 2012