Pessimistic Reward Models for Off-Policy Learning in Recommendation
Authors:
Open Access
- 13 September 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 63-74
Abstract
Methods for bandit learning from user interactions often require a model of the reward a certain context-action pair will yield – for example, the probability of a click on a recommendation. This common machine learning task is highly non-trivial, as the data-generating process for contexts and actions is often skewed by the recommender system itself. Indeed, when the deployed recommendation policy at data collection time does not pick its actions uniformly-at-random, this leads to a selection bias that can impede effective reward modelling. This in turn makes off-policy learning – the typical setup in industry – particularly challenging. In this work, we propose and validate a general pessimistic reward modelling approach for off-policy learning in recommendation. Bayesian uncertainty estimates allow us to express scepticism about our own reward model, which can in turn be used to generate a conservative decision rule. We show how it alleviates a well-known decision making phenomenon known as the Optimiser’s Curse, and draw parallels with existing work on pessimistic policy learning. Leveraging the available closed-form expressions for both the posterior mean and variance when a ridge regressor models the reward, we show how to apply pessimism effectively and efficiently to an off-policy recommendation use-case. Empirical observations in a wide range of environments show that being conservative in decision-making leads to a significant and robust increase in recommendation performance. The merits of our approach are most outspoken in realistic settings with limited logging randomisation, limited training samples, and larger action spaces.Keywords
Funding Information
- Vlaamse Overheid
This publication has 12 references indexed in Scilit:
- Policy-Aware Unbiased Learning to Rank for Top-k Rankings2020
- RecVAE: A New Variational Autoencoder for Top-N Recommendations with Implicit Feedback2020
- Recommending what video to watch next2019
- Revisiting offline evaluation for implicit-feedback recommender systems2019
- Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts2018
- Practical Lessons from Predicting Clicks on Ads at Facebook2014
- A contextual-bandit approach to personalized news article recommendation2010
- Truncated Importance SamplingJournal of Computational and Graphical Statistics, 2008
- The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision AnalysisManagement Science, 2006
- Improving predictive inference under covariate shift by weighting the log-likelihood functionJournal of Statistical Planning and Inference, 2000