The Effect of the Question Mark Option in Progress Testing: A Large-Scale Longitudinal Study
Authors:
- Elise V. Van WijkEWElise V. Van Wijk
- Jeroen DonkersJDJeroen Donkers
- Peter C. J. De LaatPLPeter C. J. De Laat
- Ariadne A. MeiboomAMAriadne A. Meiboom
- Bram JacobsBram Jacobs
- Jan Hindrik RaveslootJRJan Hindrik Ravesloot
- René A. TioRTRené A. Tio
- Frederike M. M. OudFOFrederike M. M. Oud
- André J. A. BremersABAndré J. A. Bremers
- Alexandra M. J. LangersALAlexandra M. J. Langers
Open Access
- 9 January 2025
- journal article
- Published by Ubiquity Press, Ltd. in Perspectives on Medical Education
- Vol. 14 (1) , 891-904
Abstract
Introduction: Formula scoring, widely used in medical progress tests (PT), includes a question mark option to discourage guessing, but this feature may disadvantage risk-averse students and bias results due to test-taking strategies. To enhance reliability and more accurately assess ability, Dutch medical schools recently transitioned to a computer adaptive-PT (CA-PT) based on Item Response Theory, which adjusts question difficulty dynamically, excluding the question mark option. This provided a unique opportunity to evaluate the impact of the question mark option in a large cohort. We specifically explored the relationship between question mark use in conventional PT and performance on CA-PT. Methods: Retrospective data from medical students across seven faculties who took both PT formats were analyzed. Z-scores for total score and question mark score (number of unanswered questions) in the conventional PT, and theta score for the CA-PT were assessed. A linear model assessed the effect of the question mark score on theta, corrected for the conventional PT-score. Cluster analysis explored student subgroups per year. Results: Students with similar conventional PT scores who left more questions unanswered on the conventional PT generally performed better on CA-PT. This effect diminished as students advanced through their studies. Cluster analysis revealed a variable effect between different students, most pronounced in year 4, and a reverse effect in year 5. Discussion: Question mark option use significantly impacted student performance on PT, with a remarkable variability among students. This variability suggests that formula scoring captures more than knowledge alone, highlighting the need to align scoring methods with intended assessment goals.Keywords
This publication has 33 references indexed in Scilit:
- The progress test of medicine: the Dutch experiencePerspectives on Medical Education, 2016
- The don’t know option in progress testingAdvances in Health Sciences Education, 2015
- Psychometrics Behind Computerized Adaptive TestingPsychometrika, 2015
- Lucky guess or knowledge: a cross-sectional study using the Bland and Altman analysis to compare confidence-based testing of pharmacological knowledge in 3rd and 5th year medical studentsAdvances in Health Sciences Education, 2014
- The self-consistency model of subjective confidence.Psychological Review, 2012
- The effect of a ‘don't know’ option on test scores: number‐right and formula scoring comparedMedical Education, 1999
- Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments.Journal of Personality and Social Psychology, 1999
- Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning.American Psychologist, 1995
- Weighted Likelihood Estimation of Ability in Item Response TheoryPsychometrika, 1989
- The theory of decision making.Psychological Bulletin, 1954