Combining Diverse Information for Coordinated Action: Stochastic Bandit Algorithms for Heterogeneous Agents
Authors:
Preprint
Abstract
Stochastic multi-agent multi-armed bandits typically assume that the rewards from each arm follow a fixed distribution, regardless of which agent pulls the arm. However, in many real-world settings, rewards can depend on the sensitivity of each agent to their environment. In medical screening, disease detection rates can vary by test type; in preference matching, rewards can depend on user preferences; and in environmental sensing, observation quality can vary across sensors. Since past work does not specify how to allocate agents of heterogeneous but known sensitivity of these types in a stochastic bandit setting, we introduce a UCB-style algorithm, Min-Width, which aggregates information from diverse agents. In doing so, we address the joint challenges of (i) aggregating the rewards, which follow different distributions for each agent-arm pair, and (ii) coordinating the assignments of agents to arms. Min-Width facilitates efficient collaboration among heterogeneous agents, exploiting the known structure in the agents' reward functions to weight their rewards accordingly. We analyze the regret of Min-Width and conduct pseudo-synthetic and fully synthetic experiments to study the performance of different levels of information sharing. Our results confirm that the gains to modeling agent heterogeneity tend to be greater when the sensitivities are more varied across agents, while combining more information does not always improve performance.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Multi-agent Heterogeneous Stochastic Linear Bandits2023
- Differentially Private Federated Combinatorial Bandits with Constraints2023
- Distributed Bandits with Heterogeneous Agents2022
- Efficient and targeted COVID-19 border testing via reinforcement learningNature, 2021
- A Successive-Elimination Approach to Adaptive Robotic Source SeekingIEEE Transactions on Robotics, 2020
- Multi-armed bandits in multi-agent networks2017
- Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting2014
- Combinatorial Network Optimization With Unknown Variables: Multi-Armed Bandits With Linear Rewards and Individual ObservationsIEEE/ACM Transactions on Networking, 2012
- Exploration–exploitation tradeoff using variance estimates in multi-armed banditsTheoretical Computer Science, 2009
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002