Pooled Evaluation Over Query Variations
Authors:
Open Access
- 17 October 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1759-1762
Abstract
Evaluation of information retrieval systems with test collections makes use of a suite of fixed resources: a document corpus; a set of topics; and associated judgments of the relevance of each document to each topic. With large modern collections, exhaustive judging is not feasible. Therefore an approach called pooling is typically used where, for example, the documents to be judged can be determined by taking the union of all documents returned in the top positions of the answer lists returned by a range of systems. Conventionally, pooling uses system variations to provide diverse documents to be judged for a topic; different user queries are not considered. We explore the ramifications of user query variability on pooling, and demonstrate that conventional test collections do not cover this source of variation. The effect of user query variation on the size of the judging pool is just as strong as the effect of retrieval system variation. We conclude that user query variation should be incorporated early in test collection construction, and cannot be considered effectively post hoc.Keywords
Funding Information
- Australian Research Council's Discovery Projects scheme (DP110101934, DP140102655)
This publication has 10 references indexed in Scilit:
- User Variability and IR System Evaluation2015
- A similarity measure for indefinite rankingsACM Transactions on Information Systems, 2010
- Bias and the limits of pooling for large collectionsInformation Retrieval Journal, 2007
- Strategic system comparisons via targeted relevance judgments2007
- Reliable information retrieval evaluation with incomplete and biased judgements2007
- A Markov random field model for term dependencies2005
- Question answering in TREC2001
- A probabilistic model of information retrieval: development and comparative experimentsInformation Processing & Management, 2000
- Variations in relevance judgments and the measurement of retrieval effectivenessInformation Processing & Management, 2000
- How reliable are the results of large-scale information retrieval experiments?1998