An Examination of the Replicability of Angoff Standard Setting Results Within a Generalizability Theory Framework
- 12 June 2014
- journal article
- Published by Wiley in Journal of Educational Measurement
- Vol. 51 (2) , 127-140
Abstract
Evidence of stable standard setting results over panels or occasions is an important part of the validity argument for an established cut score. Unfortunately, due to the high cost of convening multiple panels of content experts, standards often are based on the recommendation from a single panel of judges. This approach implicitly assumes that the variability across panels will be modest, but little evidence is available to support this assertion. This article examines the stability of Angoff standard setting results across panels. Data were collected for six independent standard setting exercises, with three panels participating in each exercise. The results show that although in some cases the panel effect is negligible, for four of the six data sets the panel facet represented a large portion of the overall error variance. Ignoring the often hidden panel/occasion facet can result in artificially optimistic estimates of the cut score stability. Results based on a single panel should not be viewed as a reasonable estimate of the results that would be found over multiple panels. Instead, the variability seen in a single panel can best be viewed as a lower bound of the expected variability when the exercise is replicated.Keywords
This publication has 13 references indexed in Scilit:
- The Effect of Data Format on Integration of Performance Data Into Angoff JudgmentsInternational Journal of Testing, 2013
- An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting ProcedureApplied Measurement in Education, 2008
- A Generalizability Theory Approach to Standard Error Estimates for Bookmark Standard SettingsEducational and Psychological Measurement, 2008
- Teachers' Conceptions of the Target Examinee in Angoff Standard SettingApplied Measurement in Education, 2005
- A Meta-Analysis of the Effects of Modifications to the Angoff Method on Cutoff Scores and Judgment ConsensusEducational and Psychological Measurement, 2003
- Multivariate Generalizability Analysis of the Impact of Training and Examinee Performance Information on Judgments Made in an Angoff‐Style Standard‐Setting ProcedureJournal of Educational Measurement, 2002
- Generalizability Theory2001
- Social Influences in Standard Setting: The Effects of Social Interaction on Group JudgmentsReview of Educational Research, 1989
- Errors of Measurement and Standard Setting in Mastery TestingApplied Psychological Measurement, 1984
- A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability TheoryApplied Psychological Measurement, 1980