Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
Open Access
- 19 May 2023
- journal article
- review article
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 9 (1) , e44084
Abstract
Background: Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees’ responses have to be analyzed and scored to derive information about examinees’ true knowledge. Objective: The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees’ true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived. Methods: Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees’ true knowledge using fictitious examinations with 100 single-choice items were calculated. Results: A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between –3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between –1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between –2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees’ true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between –60.0% (95% CI –60% to –60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5. Conclusions: In examinations with single-choice items, the scoring result is not always equivalent to examinees’ true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used.Keywords
All Related Versions
This publication has 170 references indexed in Scilit:
- Use of key feature questions in summative assessment of veterinary medicine studentsIrish Veterinary Journal, 2013
- The Effect of an ‘‘Unsure’’ Option on Early Childhood Professionals’ Pre- and Post-Training Knowledge AssessmentsAmerican Journal of Evaluation, 2010
- The Effects of Violating Standard Item Writing Principles on Tests and Students: The Consequences of Using Flawed Test Items on Achievement Examinations in Medical EducationAdvances in Health Sciences Education, 2005
- A Multiple Choice Test that Rewards Partial KnowledgeJournal of Further and Higher Education, 2001
- Multiple‐choice examinations: adopting an evidence‐based approach to exam techniqueAnaesthesia, 1998
- Item Type and Cognitive Ability Measured: The Validity Evidence for Multiple True-False Items in Medical Specialty CertificationApplied Measurement in Education, 1995
- To Guess or Not to Guess: A Decision‐Theoretic View of Formula ScoringJournal of Educational Measurement, 1993
- Scoring the objective structured clinical examination using a microcomputerMedical Education, 1989
- Ideal Multiple-choice ItemsJournal of the American Statistical Association, 1970
- The Rights-Minus-Wrongs Method of Correcting Chance Factors in the True-False ExaminationThe Pedagogical Seminary and Journal of Genetic Psychology, 1940