What's Wrong with Hit Ratio?
Author:
- 1 November 2006
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Intelligent Systems
- Vol. 21 (6) , 68-70
Abstract
When reporting classifier accuracy, it's common to use hit ratio as a primary metric. However, hit ratio has a serious flaw. We examine the issues surrounding this flaw and explore its magnitude through an empirical experiment on three multivalued classification data sets, using two well-known machine learning models. The results demonstrate a real problem that we can't simply overlook, and we propose an alternative-Cohen's kappa. Like any other metric, it has its own shortcomings, but we believe it should be mandatory in any scientific report about classifier accuracyKeywords
This publication has 6 references indexed in Scilit:
- Kappa and Its Dependence on Marginal Rates2005
- High agreement but low kappa: II. Resolving the paradoxesJournal of Clinical Epidemiology, 1990
- High agreement but low Kappa: I. the problems of two paradoxesJournal of Clinical Epidemiology, 1990
- A rEAPPRAISAL OF THE KAPPA COEFFICIENTJournal of Clinical Epidemiology, 1988
- MISINTERPRETATION AND MISUSE OF THE KAPPA STATISTICAmerican Journal of Epidemiology, 1987
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960