Non-Stationary Contextual Bandit Learning via Neural Predictive Ensemble Sampling
Preprint
Abstract
Real-world applications of contextual bandits often exhibit non-stationarity due to seasonality, serendipity, and evolving social trends. While a number of non-stationary contextual bandit learning algorithms have been proposed in the literature, they excessively explore due to a lack of prioritization for information of enduring value, or are designed in ways that do not scale in modern applications with high-dimensional user-specific features and large action set, or both. In this paper, we introduce a novel non-stationary contextual bandit algorithm that addresses these concerns. It combines a scalable, deep-neural-network-based architecture with a carefully designed exploration mechanism that strategically prioritizes collecting information with the most lasting value in a non-stationary environment. Through empirical evaluations on two real-world recommendation datasets, which exhibit pronounced non-stationarity, we demonstrate that our approach significantly outperforms the state-of-the-art baselines.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Kolmogorov–Smirnov Test-Based Actively-Adaptive Thompson Sampling for Non-Stationary BanditsIEEE Transactions on Artificial Intelligence, 2021
- A Change-Detection-Based Thompson Sampling Framework for Non-Stationary BanditsIEEE Transactions on Computers, 2020
- Survey on Applications of Multi-Armed and Contextual Bandits2020
- Recommendation system development for fashion retail e-commerceElectronic Commerce Research and Applications, 2018
- The non-stationary stochastic multi-armed bandit problemInternational Journal of Data Science and Analytics, 2017
- A survey of serendipity in recommender systemsKnowledge-Based Systems, 2016
- Learning in Nonstationary Environments: A SurveyIEEE Computational Intelligence Magazine, 2015
- The Nonstochastic Multiarmed Bandit ProblemSIAM Journal on Computing, 2002
- A Decision-Theoretic Generalization of On-Line Learning and an Application to BoostingJournal of Computer and System Sciences, 1997
- Asymptotically efficient adaptive allocation rulesAdvances in Applied Mathematics, 1985