Ben Scheetz
Research Subjects
Ben Scheetz has a total of 10 co-authors. Here are the top co-authors by number of shared publications:
- Anton Polishko
Google, Mountain View, CA, USA
1 shared publication - Chih-Wei Hsu
Google Inc, Mountain View, United States
1 shared publication - Craig Boutilier
Google research
1 shared publication - Hubert Pham
Google Research, Mountain View, CA, USA
1 shared publication - James Pine
Google Research, Mountain View, CA, USA
1 shared publication - Li Yang
Google research
1 shared publication
Latest Publications
- Open Access
- 10 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 2925-2929
Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
Evaluation of policies in recommender systems typically involves A/B live experiments on real users to assess a new policy's impact on relevant metrics. This "gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for onboarding users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of preference elicitation algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we can test new algorithms in a way that reliably predicts their performance on key metrics when deployed live.
- 10 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 2925-2929
Open AccessMinimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
Evaluation of policies in recommender systems typically involves A/B live experiments on real users to assess a new policy's impact on relevant metrics. This "gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for onboarding users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of preference elicitation algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we can test new algorithms in a way that reliably predicts their performance on key metrics when deployed live.