A Measure of Agreement for Interval or Nominal Multivariate Observations
Authors:
- 1 April 2001
- journal article
- research article
- Published by SAGE Publications in Educational and Psychological Measurement
- Vol. 61 (2) , 277-289
Abstract
This article addresses the problem of accounting overall chance-corrected interobserver agreement among the multivariate ratings of several judges. Modifying an approach by Berry and Mielke, a generalization of Cohen’s kappa coefficient is proposed. The generalized statistic accounts agreement for multivariate interval or nominal observations among several judges. The statistic’s metric is conventional, and in the univariate case it is equivalent to existing extensions of the kappa coefficient to several observers, as well as to a certain intraclass correlation coefficient. Application is exemplified using hypothetical and real-life data sets.Keywords
This publication has 16 references indexed in Scilit:
- An Introduction to the Bootstrap1993
- A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple RatersEducational and Psychological Measurement, 1988
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- Integration and generalization of kappas for multiple raters.Psychological Bulletin, 1980
- Inference About Weighted Kappa in the Non-Null CaseApplied Psychological Measurement, 1978
- The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of ReliabilityEducational and Psychological Measurement, 1973
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- Large sample standard errors of kappa and weighted kappa.Psychological Bulletin, 1969
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960