Evaluating Stochastic Rankings with Expected Exposure
Preprint
Abstract
We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item receive more or less expected exposure compared to any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a distribution over rankings instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including ad hoc retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Toward creating a fairer ranking in search engine resultsInformation Processing & Management, 2020
- The MovieLens DatasetsACM Transactions on Interactive Intelligent Systems, 2015
- Fast and reliable online learning to rank for information retrievalACM SIGIR Forum, 2013
- Evaluating diversified search results using per-intent graded relevance2011
- Fast als-based matrix factorization for explicit and implicit feedback datasets2010
- A general approximation framework for direct optimization of information retrieval measuresInformation Retrieval Journal, 2009
- Expected reciprocal rank for graded relevance2009
- Risky business2009
- Rank-biased precision for measurement of retrieval effectivenessACM Transactions on Information Systems, 2008
- Generating a random permutation with random transpositionsProbability Theory and Related Fields, 1981