Evaluating Stochastic Rankings with Expected Exposure
Authors:
Open Access
- 19 October 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 275-284
Abstract
We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. %Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a distribution over rankings instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including ad hoc retrieval and recommendation. %We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Measuring Fairness in Ranked Outputs2017
- Unbiased Learning-to-Rank with Biased Feedback2017
- Learning to Rank with Selection Bias in Personal Search2016
- Fairness through awareness2012
- Rank-biased precision for measurement of retrieval effectivenessACM Transactions on Information Systems, 2008
- Active exploration for learning rankings from clickthrough data2007
- Subset Ranking Using Regression2006
- Learning to rank using gradient descent2005
- Generating a random permutation with random transpositionsProbability Theory and Related Fields, 1981
- The Analysis of PermutationsJournal of the Royal Statistical Society Series C: Applied Statistics, 1975