Policy-Aware Unbiased Learning to Rank for Top-k Rankings
Authors:
Open Access
- 25 July 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 489-498
Abstract
Counterfactual Learning to Rank (LTR) methods optimize ranking systems using logged user interactions that contain interaction biases. Existing methods are only unbiased if users are presented with all relevant items in every ranking. There is currently no existing counterfactual unbiased LTR method for top-k rankings. We introduce a novel policy-aware counterfactual estimator for LTR metrics that can account for the effect of a stochastic logging policy. We prove that the policy-aware estimator is unbiased if every relevant item has a non-zero probability to appear in the top-k ranking. Our experimental results show that the performance of our estimator is not affected by the size of k: for any k, the policy-aware estimator reaches the same retrieval performance while learning from top-k feedback as when learning from feedback on the full ranking. Lastly, we introduce novel extensions of traditional LTR methods to perform counterfactual LTR and to optimize top-k metrics. Together, our contributions introduce the first policy-aware unbiased LTR approach that learns from top-k feedback and optimizes top-k metrics. As a result, counterfactual LTR is now applicable to the very prevalent top-k ranking setting in search and recommendation.Keywords
All Related Versions
Funding Information
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (612.001.551)
This publication has 22 references indexed in Scilit:
- Unbiased Learning-to-Rank with Biased Feedback2017
- Learning to Rank with Selection Bias in Personal Search2016
- Reusing historical interaction data for faster online learning to rank for IR2013
- Novelty and Diversity in Top-N Recommendation -- Analysis and EvaluationACM Transactions on Internet Technology, 2011
- Performance of recommender algorithms on top-n recommendation tasks2010
- Learning to Rank for Information RetrievalFoundations and Trends® in Information Retrieval, 2009
- Accurately interpreting clickthrough data as implicit feedback2005
- On Real-Time Top k Querying for Mobile ServicesLecture Notes in Computer Science, 2002
- Optimizing search engines using clickthrough data2002
- The Nature of Statistical Learning Theory2000