Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2925-2929
Abstract
Evaluation of policies in recommender systems typically involves A/B live experiments on real users to assess a new policy's impact on relevant metrics. This "gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for onboarding users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of preference elicitation algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we can test new algorithms in a way that reliably predicts their performance on key metrics when deployed live.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Learning from Negative User Feedback and Measuring Responsiveness for Sequential Recommenders2023
- Facebook’s Cyber–Cyber and Cyber–Physical Digital Twins2021
- Demonstrating Principled Uncertainty Modeling for Recommender Ecosystems with RecSim NG2020
- SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets2019
- A Simple Convolutional Generative Network for Next Item Recommendation2019
- Deep reinforcement learning for page-wise recommendations2018
- Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks2017
- Context-Aware Sequential Recommendation2016
- Long Short-Term MemoryNeural Computation, 1997
- Neural networks and physical systems with emergent collective computational abilities.Proceedings of the National Academy of Sciences, 1982