An Analysis of Variance Model for Assessing Reliability of Naturalistic Observations
Author:
- 1 December 1978
- journal article
- research article
- Published by SAGE Publications in Perceptual and Motor Skills
- Vol. 47 (1) , 271-278
Abstract
This paper presents a subjects × raters partially nested factorial analysis of variance model for estimating coefficients of interrater reliability. Procedures and formulae are described for computing unbiased estimates using the between-subjects and error mean squares from the model. Inclusion or exclusion of rater variance in the estimates is also discussed. Nine specific advantages for using the analysis of variance approach over existing methods are listed. A direction for future research on reliability is suggested.Keywords
This publication has 22 references indexed in Scilit:
- Utility of Analysis of Variance with Repeated Measures Programs for Estimating ReliabilityPerceptual and Motor Skills, 1975
- Weighted Chi Square: an Extension of the Kappa MethodEducational and Psychological Measurement, 1972
- A METHOD TO INTEGRATE DESCRIPTIVE AND EXPERIMENTAL FIELD STUDIES AT THE LEVEL OF DATA AND EMPIRICAL CONCEPTS1Journal of Applied Behavior Analysis, 1968
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A NEW VIEW OF INTER‐OBSERVER AGREEMENT1Personnel Psychology, 1963
- THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY†British Journal of Statistical Psychology, 1963
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- Average Values of Mean Squares in FactorialsThe Annals of Mathematical Statistics, 1956
- Estimation of the Reliability of RatingsPsychometrika, 1951
- THE COMPARISON OF PERCENTAGES IN MATCHED SAMPLESBiometrika, 1950