A Generalizability Theory Approach to Standard Error Estimates for Bookmark Standard Settings
Authors:
- 14 January 2008
- journal article
- research article
- Published by SAGE Publications in Educational and Psychological Measurement
- Vol. 68 (4) , 603-620
Abstract
The bookmark standard-setting procedure is an item response theory–based method that is widely implemented in state testing programs. This study estimates standard errors for cut scores resulting from bookmark standard settings under a generalizability theory model and investigates the effects of different universes of generalization and error sources on standard errors. This study produced several notable results. First, different patterns of variance component estimates are found for different cut scores; therefore, researchers should estimate separate variance components for each cut score and use them to estimate corresponding standard errors. Second, different universes of generalization produce different standard error estimates; thus, policy makers should consider which universe is appropriate for the proposed use of cut scores. Third, participants and groups have nonnegligible effects on several error sources. To decrease the standard errors for cut scores, increasing the number of small groups seems more efficient than increasing the number of participants.Keywords
This publication has 11 references indexed in Scilit:
- The Bookmark Standard‐Setting Method: A Literature ReviewEducational Measurement: Issues and Practice, 2006
- Generalizability Theory2001
- Models of Generalizability Theory in Analyzing Existing Faculty Evaluation DataApplied Measurement in Education, 2000
- Judgmental Item Analysis of the Nedelsky and Angoff Standard-Setting MethodsApplied Measurement in Education, 1999
- Validating the Performance Standards Associated With Passing ScoresReview of Educational Research, 1994
- A Consumer’s Guide to Setting Performance Standards on Criterion-Referenced TestsReview of Educational Research, 1986
- Errors of Measurement and Standard Setting in Mastery TestingApplied Psychological Measurement, 1984
- Standard Setting Issues and MethodsApplied Psychological Measurement, 1980
- A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability TheoryApplied Psychological Measurement, 1980
- Absolute Grading Standards for Objective TestsEducational and Psychological Measurement, 1954