Interpreting TF-IDF term weights as making relevance decisions
Top Cited Papers
Open Access
- 20 June 2008
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 26 (3) , 1-37
Abstract
A novel probabilistic retrieval model is presented. It forms a basis to interpret the TF-IDF term weights as making relevance decisions. It simulates the local relevance decision-making for every location of a document, and combines all of these “local” relevance decisions as the “document-wide” relevance decision for the document. The significance of interpreting TF-IDF in this way is the potential to: (1) establish a unifying perspective about information retrieval as relevance decision-making; and (2) develop advanced TF-IDF-related term weights for future elaborate retrieval models. Our novel retrieval model is simplified to a basic ranking formula that directly corresponds to the TF-IDF term weights. In general, we show that the term-frequency factor of the ranking formula can be rendered into different term-frequency factors of existing retrieval systems. In the basic ranking formula, the remaining quantity - log p (r¯| t ∈ d ) is interpreted as the probability of randomly picking a nonrelevant usage (denoted by r¯) of term t . Mathematically, we show that this quantity can be approximated by the inverse document-frequency (IDF). Empirically, we show that this quantity is related to IDF, using four reference TREC ad hoc retrieval data collections.Keywords
Funding Information
- Research Grants Council, University Grants Committee, Hong Kong (PolyU 5226/05E)
This publication has 58 references indexed in Scilit:
- Local versus global link information in the WebACM Transactions on Information Systems, 2003
- Efficient passage ranking for document databasesACM Transactions on Information Systems, 1999
- “Is this document relevant?…probably”ACM Computing Surveys, 1998
- Overview of the Okapi projectsJournal of Documentation, 1997
- Foundations of Probabilistic and Utility-Theoretic IndexingJournal of the ACM, 1978
- THE PROBABILITY RANKING PRINCIPLE IN IRJournal of Documentation, 1977
- Precision Weighting—An Effective Automatic Indexing MethodJournal of the ACM, 1976
- A statistical analysis of interdependence in character sequencesInformation Sciences, 1975
- A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVALJournal of Documentation, 1972
- An experiment in automatic indexingAmerican Documentation, 1965