Detecting Nonadditivity in Single‐Facet Generalizability Theory Applications: Tukey's Test
Authors:
- 1 March 2018
- journal article
- research article
- Published by Wiley in Journal of Educational Measurement
- Vol. 55 (1) , 78-89
Abstract
Under the generalizability‐theory (G‐theory) framework, the estimation precision of variance components (VCs) is of significant importance in that they serve as the foundation of estimating reliability. Zhang and Lin advanced the discussion of nonadditivity in data from a theoretical perspective and showed the adverse effects of nonadditivity on the estimation precision of VCs in 2016. Contributing to this line of research, the current article directs the discussion of nonadditivity from a theoretical perspective to a practical application and highlights the importance of detecting nonadditivity in G‐theory applications. To this end, Tukey's test for nonadditivity is the only method to date that is appropriate for the typical single‐facet G‐theory design, in which a single observation is made per element within a facet. The current article evaluates the Type I and Type II error rates of Tukey's test. Results show that Tukey's test is satisfactory in controlling for falsely detecting nonadditivity when the data are actually additive and that it is generally powerful in detecting nonadditivity when it exists. Finally, the article demonstrates an application of Tukey's test in detecting nonadditivity in a judgmental study of educational standards and shows how Tukey's test results can be used to correct imprecision in the estimated VC in the presence of nonadditivity.Keywords
This publication has 19 references indexed in Scilit:
- Score generalizability of academic writing tasks: Does one test method fit it all?Language Testing, 2009
- An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting ProcedureApplied Measurement in Education, 2008
- Evaluating Prototype Tasks and Alternative Rating Schemes for a New ESL Writing Test through G-theoryInternational Journal of Testing, 2007
- Estimating Standard Errors of Cut Scores for Item Rating and Mapmark ProceduresEducational and Psychological Measurement, 2007
- Performance Assessments from the Perspective of Generalizability TheoryApplied Psychological Measurement, 2000
- Generalizability Analyses of Work Keys Listening and Writing TestsEducational and Psychological Measurement, 1995
- Sampling Variability of Performance AssessmentsJournal of Educational Measurement, 1993
- A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability TheoryApplied Psychological Measurement, 1980
- The Examination and Analysis of ResidualsTechnometrics, 1963
- One Degree of Freedom for Non-Additivity1949