Evaluation Measures for Ordinal Regression
Authors:
- Stefano BaccianellaSBStefano Baccianella
- Andrea EsuliAEAndrea Esuli
- Fabrizio SebastianiFSFabrizio Sebastiani
- 1 January 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21647143,p. 283-287
Abstract
Ordinal regression (OR-also known as ordinal classification) has received increasing attention in recent times, due to its importance in IR applications such as learning to rank and product review rating. However, research has not paid attention to the fact that typical applications of OR often involve datasets that are highly imbalanced. An imbalanced dataset has the consequence that, when testing a system with an evaluation measure conceived for balanced datasets, a trivial system assigning all items to a single class (typically, the majority class) may even outperform genuinely engineered systems. Moreover, if this evaluation measure is used for parameter optimization, a parameter choice may result that makes the system behave very much like a trivial system. In order to avoid this, evaluation measures that can handle imbalance must be used. We propose a simple way to turn standard measures for OR into ones robust to imbalance. We also show that, once used on balanced datasets, the two versions of each measure coincide, and therefore argue that our measures should become the standard choice for OR.Keywords
This publication has 17 references indexed in Scilit:
- Opinion Mining and Sentiment AnalysisFoundations and Trends® in Information Retrieval, 2008
- Seeing Several Stars: A Rating Inference Task for a Document Containing Several Evaluation Criteria2008
- Opinion spam and analysis2008
- Seeing stars when there aren't many stars2006
- Seeing stars2005
- Extracting product features and opinions from reviews2005
- EditorialACM SIGKDD Explorations Newsletter, 2004
- The sentimental factor2004
- Machine learning in automated text categorizationACM Computing Surveys, 2002
- An Evaluation of Statistical Approaches to Text CategorizationInformation Retrieval Journal, 1999