User Response Models to Improve a REINFORCE Recommender System
Authors:
Open Access
- 8 March 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 121-129
Abstract
Reinforcement Learning (RL) techniques have been sought after as the next-generation tools to further advance the field of recommendation research. Different from classic applications of RL, recommender agents, especially those deployed on commercial recommendation platforms, have to operate in extremely large state and action spaces, serving a dynamic user base in the order of billions, and a long-tail item corpus in the order of millions or billions. The (positive) user feedback available to train such agents is extremely scarce in retrospect. Improving the sample efficiency of RL algorithms is thus of paramount importance when developing RL agents for recommender systems. In this work, we present a general framework to augment the training of model-free RL agents with auxiliary tasks for improved sample efficiency. More specifically, we opt to add additional tasks that predict users' immediate responses (positive or negative) toward recommendations, i.e., user response modeling, to enhance the learning of the state and action representations for the recommender agents. We also introduce a tool based on gradient correlation analysis to guide the model design. We showcase the efficacy of our method in offline experiments, learning and evaluating agent policies over hundreds of millions of user trajectories. We also conduct live experiments on an industrial recommendation platform serving billions of users and tens of millions of items to verify its benefit.Keywords
This publication has 13 references indexed in Scilit:
- Reinforcement Learning to Rank in E-Commerce Search Engine2018
- Time-Contrastive Networks: Self-Supervised Learning from Video2018
- Offline A/B Testing for Recommender Systems2018
- Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition2017
- Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collectionThe International Journal of Robotics Research, 2017
- Overcoming catastrophic forgetting in neural networksProceedings of the National Academy of Sciences, 2017
- Deep Neural Networks for YouTube Recommendations2016
- Matrix Factorization Techniques for Recommender SystemsComputer, 2009
- Batch reinforcement learning in a complex domain2007
- Item-based collaborative filtering recommendation algorithms2001