A Multivariate Generalizability Theory Approach to Standard Setting
Authors:
- 8 April 2015
- journal article
- research article
- Published by SAGE Publications in Applied Psychological Measurement
- Vol. 39 (7) , 507-524
Abstract
Generalizability theory (G theory) allows researchers to assess the many sources of variance inherent in complex standard setting procedures involving the determination of cut scores. The flexibility of G and D studies provides a way to conceptualize and quantify the results of different standard settings once the universe of admissible observations and the universe of generalization are defined. The current article applies a multivariate single-facet design for estimating standard errors of cut scores. For practical purposes, several multivariate D study designs are used to investigate what effect various panel sizes and test lengths have on the precision of the standard setting process. The current study demonstrates the advantages and usefulness of multivariate G theory in determining the accuracy of cut scores in practical applications of standard setting procedures.Keywords
This publication has 15 references indexed in Scilit:
- An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting ProcedureApplied Measurement in Education, 2008
- A Generalizability Theory Approach to Standard Error Estimates for Bookmark Standard SettingsEducational and Psychological Measurement, 2008
- Estimating Standard Errors of Cut Scores for Item Rating and Mapmark ProceduresEducational and Psychological Measurement, 2007
- Comparison of a rational and an empirical standard setting procedure for an OSCEMedical Education, 2003
- A Response to “Setting Reasonable and Useful Performance Standards” in the National Academy of Science’Grading the Nations Report CardEducational Measurement: Issues and Practice, 2000
- Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judgesMedical Education, 1999
- Judgmental Item Analysis of the Nedelsky and Angoff Standard-Setting MethodsApplied Measurement in Education, 1999
- Errors of Measurement and Standard Setting in Mastery TestingApplied Psychological Measurement, 1984
- A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability TheoryApplied Psychological Measurement, 1980
- Differential Weighting: A Review of Methods and Empirical StudiesReview of Educational Research, 1970