Liberal relevance criteria of TREC -
Author:
Open Access
- 11 August 2002
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 324-330
Abstract
Most test collections (like TREC and CLEF) for experimental research in information retrieval apply binary relevance assessments. This paper introduces a four-point relevance scale and reports the findings of a project in which TREC-7 and TREC-8 document pools on 38 topics were reassessed. The goal of the reassessment was to build a subcollection of TREC for experiments on highly relevant documents and to learn about the assessment process as well as the characteristics of a multigraded relevance corpus.Relevance criteria were defined so that a distinction was made between documents rich in topical information (relevant and highly relevant documents) and poor in topical information (marginally relevant documents). It turned out that about 50% of documents assessed as relevant were regarded as marginal. The characteristics of the relevance corpus and lessons learned from the reassessment project are discussed. The need to develop more elaborated relevance assessment schemes is emphasized.Keywords
This publication has 10 references indexed in Scilit:
- Evaluation by highly relevant documents2001
- Document text characteristics affect the ranking of the most relevant documents by expanded structured queriesJournal of Documentation, 2001
- Variations in relevance judgments and the measurement of retrieval effectivenessInformation Processing & Management, 2000
- IR evaluation methods for retrieving highly relevant documents2000
- Finding information on the World Wide Web: the retrieval effectiveness of search enginesInformation Processing & Management, 1999
- Variations in relevance judgments and the measurement of retrieval effectiveness1998
- How reliable are the results of large-scale information retrieval experiments?1998
- Variations in relevance judgments and the evaluation of retrieval performanceInformation Processing & Management, 1992
- A study of information seeking and retrieving. I. Background and methodologyJournal of the American Society for Information Science, 1988
- An evaluation of retrieval effectiveness for a full-text document-retrieval systemCommunications of the ACM, 1985