Variance Estimation of Nominal-Scale Inter-Rater Reliability with Random Selection of Raters
Author:
- 1 September 2008
- journal article
- Published by Cambridge University Press (CUP) in Psychometrika
- Vol. 73 (3) , 407-430
Abstract
Most inter-rater reliability studies using nominal scales suggest the existence of two populations of inference: the population of subjects (collection of objects or persons to be rated) and that of raters. Consequently, the sampling variance of the inter-rater reliability coefficient can be seen as a result of the combined effect of the sampling of subjects and raters. However, all inter-rater reliability variance estimators proposed in the literature only account for the subject sampling variability, ignoring the extra sampling variance due to the sampling of raters, even though the latter may be the biggest of the variance components. Such variance estimators make statistical inference possible only to the subject universe. This paper proposes variance estimators that will make it possible to infer to both universes of subjects and raters. The consistency of these variance estimators is proved as well as their validity for confidence interval construction. These results are applicable only to fully crossed designs where each rater must rate each subject. A small Monte Carlo simulation study is presented to demonstrate the accuracy of large-sample approximations on reasonably small samples.Keywords
This publication has 32 references indexed in Scilit:
- Including Omission Mistakes in the Calculation of Cohen's Kappa and an Analysis of the Coefficient's Paradox FeaturesEducational and Psychological Measurement, 2006
- A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of JudgesEducational and Psychological Measurement, 2004
- Kappa coefficients in medical researchStatistics in Medicine, 2002
- Statistical Inferences For Interobserver Agreement Studies With Nominal Outcome DataJournal of the Royal Statistical Society: Series D (The Statistician), 2001
- Latent class analysis of diagnostic agreementStatistics in Medicine, 1990
- Survey Design under the Regression Superpopulation ModelJournal of the American Statistical Association, 1982
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- Measures of response agreement for qualitative data: Some generalizations and alternatives.Psychological Bulletin, 1971
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960