Bandits and Experts in Metric Spaces
Authors:
- Aleksandrs SlivkinsASAleksandrs Slivkins
Preprint
Abstract
In a multi-armed bandit problem, an online algorithm chooses from a set of strategies in a sequence of trials so as to maximize the total payoff of the chosen strategies. While the performance of bandit algorithms with a small finite strategy set is quite well understood, bandit problems with large strategy sets are still a topic of very active investigation, motivated by practical applications such as online auctions and web advertisement. The goal of such research is to identify broad and natural classes of strategy sets and payoff functions which enable the design of efficient solutions. In this work we study a very general setting for the multi-armed bandit problem in which the strategies form a metric space, and the payoff function satisfies a Lipschitz condition with respect to the metric. We refer to this problem as the "Lipschitz MAB problem". We present a solution for the multi-armed bandit problem in this setting. That is, for every metric space we define an isometry invariant which bounds from below the performance of Lipschitz MAB algorithms for this metric space, and we present an algorithm which comes arbitrarily close to meeting this bound. Furthermore, our technique gives even better results for benign payoff functions. We also address the full-feedback ("best expert") version of the problem, where after every round the payoffs from all arms are revealed.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Multi‐Armed Bandit Allocation Indices2011
- UCB revisited: Improved regret bounds for the stochastic multi-armed bandit problemPeriodica Mathematica Hungarica, 2010
- Exploration–exploitation tradeoff using variance estimates in multi-armed banditsTheoretical Computer Science, 2009
- Online linear optimization and adaptive routingJournal of Computer and System Sciences, 2008
- A metric for distributions with applications to image databases2002
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002
- Lectures on Analysis on Metric Spaces2001
- A Game of Prediction with Expert AdviceJournal of Computer and System Sciences, 1998
- Empirical Support for Winnow and Weighted-Majority Algorithms: Results on a Calendar Scheduling DomainMachine Learning, 1997
- Some aspects of the sequential design of experimentsBulletin of the American Mathematical Society, 1952