Variability of Estimated Variance Components and Related Statistics in a Performance Assessment
Authors:
- Xiaohong GaoXGXiaohong Gao
- Robert L. BrennanRBRobert L. Brennan
- 1 April 2001
- journal article
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 14 (2) , 191-203
Abstract
Generalizability theory provides a conceptual and statistical framework for estimating variance components and measurement precision. The theory has been widely used in evaluating technical qualities of performance assessments. However, estimates of variance components, measurement error variances, and generalizability coefficients are likely to vary from one sample to another. This study empirically investigates sampling variability of estimated variance components using data collected in several years for a listening and writing performance assessment. This study also evaluates stability of estimated measurement precision from year to year. The results indicated that the estimated variance components varied from one study to another, especially when sample sizes were small. The estimated measurement error variances and generalizability coefficients also changed from one year to another. Measurement precision projected by a generalizability study may not be fully realized in an actual decision study. The...Keywords
This publication has 6 references indexed in Scilit:
- Generalizability and Validity of a Mathematics Performance AssessmentJournal of Educational Measurement, 1996
- Generalizability Analyses of Work Keys Listening and Writing TestsEducational and Psychological Measurement, 1995
- Generalizability of Large-Scale Performance Assessments in Science: Promises and ProblemsApplied Measurement in Education, 1994
- Sampling Variability of Performance AssessmentsJournal of Educational Measurement, 1993
- ACT research report series: The bootstrap and other procedures for examining the variability of estimated variance components in testing contexts1987
- Sampling Errors of Variance Components in Small Sample Multifacet Generalizability StudiesJournal of Educational Statistics, 1978