Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods (Preprint)
Preprint
- 25 October 2022
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Scoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data. OBJECTIVE This study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6). METHODS Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05). RESULTS We analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977). CONCLUSIONS Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees’ scores and overall examination results.Keywords
All Related Versions
- Published version: JMIR Medical Education, 9 (1), e43792.
This publication has 10 references indexed in Scilit:
- Relation between examinees’ true knowledge and examination scores: systematic review and exemplary calculations on Pick-N itemsEducational Research Review, 2022
- Relation between examinees’ true knowledge and examination scores: systematic review and exemplary calculations on Multiple-True-False itemsEducational Research Review, 2021
- Effect of different scoring approaches upon credit assignment when using Multiple True‐False items in dental undergraduate examinationsEuropean Journal of Dental Education, 2018
- Gutachten zur Erstellung „gerichtsfester” Multiple-Choice-PrüfungsaufgabenPsychologische Rundschau, 2014
- Pick-N multiple choice-exams: a comparison of scoring algorithmsAdvances in Health Sciences Education, 2010
- USING MULTIPLE RESPONSE TRUE–FALSE MULTIPLE CHOICE QUESTIONSAnz Journal of Surgery, 1999
- Multiple True-False Items: A Study of Interitem Correlations, Scoring Alternatives, and Reliability EstimationJournal of Educational Measurement, 1988
- The Merits of Multiple-Answer Items as Evaluated by Using Six Scoring FormulasThe Journal of Experimental Education, 1984
- Multiple-Answer Multiple-Choice Test Items: Responding and Scoring Through Bayes and Minimax StrategiesPsychometrika, 1978