How accurate are subjective judgements of a continuum?
Authors:
- 28 February 2003
- journal article
- Published by Wiley in Histopathology
- Vol. 42 (3) , 227-232
Abstract
Aims: To assess the accuracy of subjective judgements of a uni‐dimensional continuum.Methods and results: Twenty‐four observers were asked to estimate the position of a dot placed between two lines using a 100‐point scale. Each observer judged 20 different positions. Ninety‐five percent of observations were between −7 and +9 of the true value. The median and the mean value of the estimates for each position were within one or two points of the true value in most cases with a maximum deviation of five. There was perfect correlation between the ranks of the median and the mean and the true value (correlation coefficient +1, P < 0.00001). The κ values for interobserver agreement varied from 0.98 for two categories to 0.55 for 10 categories and 0.39 for 20 categories. Information transmitted increased from 0.98 bits with two categories to 2.83 bits with 20 categories.Conclusions: The experiment indicates that subjective judgements can be accurate but it is necessary to use a large number of categories. One hundred‐point scales allow confidence intervals to be calculated and the intervals are reduced if the results from several observers are pooled. The mean and the median of the judgements of a large number of observers are close to objective reality and this approach could be used to define a gold standard of rankings of pathological continua such as cervical dyskaryosis or tumour differentiation. There is a reciprocal relationship between κ values and information transmitted which demonstrates the fallacy of using κ statistics in studies of continua.Keywords
This publication has 12 references indexed in Scilit:
- Observer accuracy in estimating proportions in images: implications for the semiquantitative assessment of staining reactions and a proposal for a new systemJournal of Clinical Pathology, 2001
- Information and observer disagreement in histopathologyHistopathology, 1994
- Sources of interobserver variation in histopathological grading of cervical dysplasiaJournal of Clinical Epidemiology, 1992
- Observer variation in histopathological diagnosis and grading of cervical intraepithelial neoplasia.BMJ, 1989
- Observer variation in the assessment of dysplasia in ulcerative colitisHistopathology, 1988
- Assessment of dysplasia in colorectal adenomas: an observer variation and morphometric study.Journal of Clinical Pathology, 1985
- Observer variation in the histological grading of rectal carcinoma.Journal of Clinical Pathology, 1983
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948