Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices
Top Cited Papers
- 1 January 2015
- journal article
- Published by Hogrefe Publishing Group in Methodology
- Vol. 11 (1) , 13-22
Abstract
Although intercoder reliability has been considered crucial to the validity of a content study, the choice among them has been controversial. This study analyzed all the content studies published in the two major communication journals that reported intercoder reliability, aiming to find how scholars conduct the intercoder reliability test. The results revealed that some intercoder reliability indices were misused persistently concerning the levels of measurement, the number of coders, and the means of reporting reliability over the past 30 years. Implications of misuse, disuse, and abuse were discussed, and suggestions regarding proper choice of indices in various situations were made at last.Keywords
This publication has 40 references indexed in Scilit:
- Inter-Coder Agreement for Computational LinguisticsComputational Linguistics, 2008
- Measuring agreement in method comparison studiesStatistical Methods in Medical Research, 1999
- Bias, prevalence and kappaJournal of Clinical Epidemiology, 1993
- High agreement but low kappa: II. Resolving the paradoxesJournal of Clinical Epidemiology, 1990
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- Integration and generalization of kappas for multiple raters.Psychological Bulletin, 1980
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin, 1959
- Communications Through Limited Response QuestioningPublic Opinion Quarterly, 1954