Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
Open Access
- 9 November 2017
- journal article
- research article
- Published by Springer Nature in BMC Medical Education
- Vol. 17 (1) , 192
Abstract
Progress testing is an assessment tool used to periodically assess all students at the end-of-curriculum level. Because students cannot know everything, it is important that they recognize their lack of knowledge. For that reason, the formula-scoring method has usually been used. However, where partial knowledge needs to be taken into account, the number-right scoring method is used. Research comparing both methods has yielded conflicting results. As far as we know, in all these studies, Classical Test Theory or Generalizability Theory was used to analyze the data. In contrast to these studies, we will explore the use of the Rasch model to compare both methods. A 2 × 2 crossover design was used in a study where 298 students from four medical schools participated. A sample of 200 previously used questions from the progress tests was selected. The data were analyzed using the Rasch model, which provides fit parameters, reliability coefficients, and response option analysis. The fit parameters were in the optimal interval ranging from 0.50 to 1.50, and the means were around 1.00. The person and item reliability coefficients were higher in the number-right condition than in the formula-scoring condition. The response option analysis showed that the majority of dysfunctional items emerged in the formula-scoring condition. The findings of this study support the use of number-right scoring over formula scoring. Rasch model analyses showed that tests with number-right scoring have better psychometric properties than formula scoring. However, choosing the appropriate scoring method should depend not only on psychometric properties but also on self-directed test-taking strategies and metacognitive skills.Keywords
Funding Information
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (9568-13-1)
This publication has 33 references indexed in Scilit:
- The use of progress testingPerspectives on Medical Education, 2012
- Optimal correction for guessing in multiple-choice testsJournal of Mathematical Psychology, 2010
- A primer on classical test theory and item response theory for assessments in medical educationMedical Education, 2009
- Evidence of gender bias in True-False-Abstain medical examinationsBMC Medical Education, 2009
- Differences in knowledge development exposed by multi-curricular progress test dataAdvances in Health Sciences Education, 2007
- To Guess or Not to Guess: A Decision‐Theoretic View of Formula ScoringJournal of Educational Measurement, 1993
- Setting Standards for Multiple-Choice Items in Clinical ReasoningEvaluation & the Health Professions, 1992
- The Projected Impact of the Correction for Guessing on Individual ScoresJournal of Educational Measurement, 1988
- Item Discrimination: When More Is WorseJournal of Educational Measurement, 1988
- The Correction for GuessingReview of Educational Research, 1973