Two-Stage Constrained Actor-Critic for Short Video Recommendation
Open Access
- 30 April 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 865-875
Abstract
The wide popularity of short videos on social media poses new opportunities and challenges to optimize recommender systems on the video-sharing platforms. Users sequentially interact with the system and provide complex and multi-faceted responses, including WatchTime and various types of interactions with multiple videos. On the one hand, the platforms aim at optimizing the users’ cumulative WatchTime (main goal) in the long term, which can be effectively optimized by Reinforcement Learning. On the other hand, the platforms also need to satisfy the constraint of accommodating the responses of multiple user interactions (auxiliary goals) such as Like, Follow, Share, etc. In this paper, we formulate the problem of short video recommendation as a Constrained Markov Decision Process (CMDP). We find that traditional constrained reinforcement learning algorithms fail to work well in this setting. We propose a novel two-stage constrained actor-critic method: At stage one, we learn individual policies to optimize each auxiliary signal. In stage two, we learn a policy to (i) optimize the main signal and (ii) stay close to policies learned in the first stage, which effectively guarantees the performance of this main policy on the auxiliaries. Through extensive offline evaluations, we demonstrate the effectiveness of our method over alternatives in both optimizing the main goal as well as balancing the others. We further show the advantage of our method in live experiments of short video recommendations, where it significantly outperforms other baselines in terms of both WatchTime and interactions. Our approach has been fully launched in the production system to optimize user experiences on the platform.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- A pareto-efficient algorithm for multiple objective optimization in e-commerce recommendation2019
- Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems2019
- Reinforcement Knowledge Graph Reasoning for Explainable Recommendation2019
- Large-Scale Interactive Recommendation with Tree-Structured Policy GradientProceedings of the AAAI Conference on Artificial Intelligence, 2019
- A Deep Reinforcement Learning Approach to Proactive Content Pushing and Recommendation for Mobile UsersIEEE Access, 2019
- Top-K Off-Policy Correction for a REINFORCE Recommender System2019
- Stabilizing Reinforcement Learning in Dynamic Environment with Application to Online Recommendation2018
- Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning2018
- Wide & Deep Learning for Recommender Systems2016
- Joint multi-grain topic sentiment: modeling semantic aspects for online reviewsInformation Sciences, 2016