Evaluating multi-query sessions
Open Access
- 24 July 2011
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1053-1062
Abstract
The standard system-based evaluation paradigm has focused on assessing the performance of retrieval systems in serving the best results for a single query. Real users, however, often begin an interaction with a search engine with a sufficiently under-specified query that they will need to reformulate before they find what they are looking for. In this work we consider the problem of evaluating retrieval systems over test collections of multi-query sessions. We propose two families of measures: a model-free family that makes no assumption about the user's behavior over a session, and a model-based family with a simple model of user interactions over the session. In both cases we generalize traditional evaluation metrics such as average precision to multi-query session evaluation. We demonstrate the behavior of the proposed metrics by using the new TREC 2010 Session track collection and simulations over the TREC-9 Query track collection.Keywords
This publication has 11 references indexed in Scilit:
- Expected reciprocal rank for graded relevance2009
- Click-based evidence for decaying weight distributions in search effectiveness metricsInformation Retrieval Journal, 2009
- Modeling Expected Utility of Multi-session Information Distillation2009
- Rank-biased precision for measurement of retrieval effectivenessACM Transactions on Information Systems, 2008
- Semantic components enhance retrieval of domain-specific documents2007
- Alternatives to Bpref2007
- Estimating average precision with incomplete and imperfect judgments2006
- Vox populi: The public searching of the webJournal of the American Society for Information Science and Technology, 2001
- The design of browsing and berrypicking techniques for the online search interfaceOnline Review, 1989
- Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systemsAmerican Documentation, 1968