Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
Top Cited Papers
Open Access
- 18 July 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 238-248
Abstract
Offline reinforcement learning (RL), a technology that offline learns a policy from logged data without the need to interact with online environments, has become a favorable choice in decision-making processes like interactive recommendation. Offline RL faces the value overestimation problem. To address it, existing methods employ conservatism, e.g., by constraining the learned policy to be close to behavior policies or punishing the rarely visited state-action pairs. However, when applying such offline RL to recommendation, it will cause a severe Matthew effect, i.e., the rich get richer and the poor get poorer, by promoting popular items or categories while suppressing the less popular ones. It is a notorious issue that needs to be addressed in practical recommender systems. In this paper, we aim to alleviate the Matthew effect in offline RL-based recommendation. Through theoretical analyses, we find that the conservatism of existing methods fails in pursuing users' long-term satisfaction. It inspires us to add a penalty term to relax the pessimism on states with high entropy of the logging policy and indirectly penalizes actions leading to less diverse states. This leads to the main technical contribution of the work: Debiased model-based Offline RL (DORL) method. Experiments show that DORL not only captures user interests well but also alleviates the Matthew effect. The implementation is available via https://github.com/chongminggao/DORL-codesKeywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61972372, U19A2079, 62121002)
- National Key Research and Development Program of China (2021YFF0901603)
This publication has 31 references indexed in Scilit:
- DGCN: Diversified Recommendation with Graph Convolutional Networks2021
- Disentangling User Interest and Conformity for Recommendation with Causal Embedding2021
- Shifting Consumption towards Diverse Content on Music Streaming Platforms2021
- Keeping Dataset Biases out of the Simulation2020
- Algorithmic Effects on the Diversity of Consumption on Spotify2020
- Pseudo Dyna-Q2020
- Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems2019
- Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding2018
- Offline A/B Testing for Recommender Systems2018
- The LFM-1b Dataset for Music Retrieval and Recommendation2016