Quantifying trends accurately despite classifier error and class imbalance
Author:
Open Access
- 20 August 2006
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 157-166
Abstract
This paper promotes a new task for supervised machine learning research: quantification - the pursuit of learning methods for accurately estimating the class distribution of a test set, with no concern for predictions on individual cases. A variant for cost quantification addresses the need to total up costs according to categories predicted by imperfect classifiers. These tasks cover a large and important family of applications that measure trends over time.The paper establishes a research methodology, and uses it to evaluate several proposed methods that involve selecting the classification threshold in a way that would spoil the accuracy of individual classifications. In empirical tests, Median Sweep methods show outstanding ability to estimate the class distribution, despite wide disparity in testing and training conditions. The paper addresses shifting class priors and costs, but not concept drift in general.Keywords
This publication has 6 references indexed in Scilit:
- Pragmatic text mining2006
- Discovering evolutionary theme patterns from text2005
- KBA: kernel boundary alignment considering imbalanced data distributionIEEE Transactions on Knowledge and Data Engineering, 2005
- Counting Positives Accurately Despite Inaccurate Classification2005
- ThemeRiver: visualizing thematic changes in large document collectionsIEEE Transactions on Visualization and Computer Graphics, 2002
- Adjusting the Outputs of a Classifier to New aPrioriProbabilities: A Simple ProcedureNeural Computation, 2002