Consistency of Angoff-Based Standard-Setting Judgments: Are Item Judgments and Passing Scores Replicable Across Different Panels of Experts?
- 2 January 2015
- journal article
- research article
- Published by Taylor & Francis in Educational Assessment
- Vol. 20 (1) , 66-78
Abstract
Angoff-based standard setting is widely used, especially for high-stakes licensure assessments. Nonetheless, some critics have claimed that the judgment task is too cognitively complex for panelists, whereas others have explicitly challenged the consistency in (replicability of) standard-setting outcomes. Evidence of consistency in item judgments and passing scores is necessary to justify using the passing scores for consequential decisions. Few studies, however, have directly evaluated consistency across different standard-setting panels. The purpose of this study was to investigate consistency of Angoff-based standard-setting judgments and passing scores across 9 different educator licensure assessments. Two independent, multistate panels of educators were formed to recommend the passing score for each assessment, with each panel engaging in 2 rounds of judgments. Multiple measures of consistency were applied to each round of judgments. The results provide positive evidence of the consistency in judgments and passing scores.Keywords
This publication has 19 references indexed in Scilit:
- The Effect of Data Format on Integration of Performance Data Into Angoff JudgmentsInternational Journal of Testing, 2013
- Judges' Use of Examinee Performance Data in an Angoff Standard‐Setting Exercise for a Medical Licensing Examination: An Experimental StudyJournal of Educational Measurement, 2009
- When Adaptation Is Not an Option: An Application of Multilingual Standard SettingJournal of Educational Measurement, 2008
- Certification Testing as an Illustration of Argument-Based ValidationMeasurement: Interdisciplinary Research and Perspectives, 2004
- More Unintended Consequences of High‐Stakes TestingEducational Measurement: Issues and Practice, 2001
- How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory StudyEducational and Psychological Measurement, 1999
- Relations Between Observed Item Difficulty Levels and Angoff Minimum Passing Levels for a Group of Borderline ExamineesApplied Measurement in Education, 1999
- Validating the Performance Standards Associated With Passing ScoresReview of Educational Research, 1994
- Standard Error of MeasurementEducational Measurement: Issues and Practice, 1991
- A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability TheoryApplied Psychological Measurement, 1980