Evaluating the Operational Feasibility of Using Subsets of Items to Recommend Minimal Competency Cut Scores
- 27 July 2015
- journal article
- research article
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 28 (4) , 292-307
Abstract
Establishing cut scores using the Angoff method requires panelists to evaluate every item on a test and make a probability judgment. This can be time-consuming when there are large numbers of items on the test. Previous research using resampling studies suggest that it is possible to recommend stable Angoff-based cut score estimates using a content-stratified subset of ˜45 items. Recommendations from earlier work were directly applied in this study in two operational standard-setting meetings. Angoff cut scores from two panels of raters were collected at each study, wherein one panel established the cut score based on the entire test, and another comparable panel first used a proportionally stratified subset of 45 items, and subsequently used the entire test in recommending the cut scores. The cut scores recommended for the subset of items were compared to the cut scores recommended based on the entire test for the same panel, and a comparable independent panel. Results from both studies suggest that cut scores recommended using a subset of items are comparable (i.e., within one standard error) to the cut score estimates from the full test.Keywords
This publication has 14 references indexed in Scilit:
- Evaluation of the Standard Setting on the 2005 Grade 12 National Assessment of Educational Progress Mathematics TestApplied Measurement in Education, 2009
- Item Selection Strategy for Reducing the Number of Items Rated in an Angoff Standard Setting StudyEducational and Psychological Measurement, 2007
- The Use of Subsets of Test Questions in an Angoff Standard-Setting MethodEducational and Psychological Measurement, 2005
- Ability of Panelists to Estimate Item Performance for a Target Group of Candidates: An Issue in Judgmental Standard SettingEducational Assessment, 2001
- How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory StudyEducational and Psychological Measurement, 1999
- Relations Between Observed Item Difficulty Levels and Angoff Minimum Passing Levels for a Group of Borderline ExamineesApplied Measurement in Education, 1999
- Validating the Performance Standards Associated With Passing ScoresReview of Educational Research, 1994
- The Reproducibility of Standards Over Groups and OccasionsApplied Measurement in Education, 1992
- The Effect of Numbers of Experts and Common Items on Cutting Score Equivalents Based on Expert JudgmentApplied Psychological Measurement, 1991
- A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability TheoryApplied Psychological Measurement, 1980