Learning Personalized Decision Support Policies
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (13) , 14203-14211
Abstract
Individual human decision-makers may benefit from different forms of support to improve decision outcomes, but when will each form of support yield better outcomes? In this work, we posit that personalizing access to decision support tools can be an effective mechanism for instantiating the appropriate use of AI assistance. Specifically, we propose the general problem of learning a decision support policy that, for a given input, chooses which form of support to provide to decision-makers for whom we initially have no prior information. We develop Modiste, an interactive tool to learn personalized decision support policies. Modiste leverages stochastic contextual bandit techniques to personalize a decision support policy for each decision-maker. In our computational experiments, we characterize the expertise profiles of decision-makers for whom personalized policies will outperform offline policies, including population-wide baselines. Our experiments include realistic forms of support (e.g., expert consensus and predictions from a large language model) on vision and language tasks. Our human subject experiments add nuance to and bolster our computational experiments, demonstrating the practical utility of personalized policies when real users benefit from accessing support across tasks.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Building machines that learn and think with peopleNature Human Behaviour, 2024
- Evaluating language models for mathematics through interactionsProceedings of the National Academy of Sciences, 2024
- Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with ExplanationsProceedings of the ACM on Human-Computer Interaction, 2023
- Online Decision Making with High-Dimensional CovariatesOperations Research, 2020
- The Principles and Limits of Algorithm-in-the-Loop Decision MakingProceedings of the ACM on Human-Computer Interaction, 2019
- Prolific.ac—A subject pool for online experimentsJournal of Behavioral and Experimental Finance, 2018
- Consensus Recommendations for Systematic Evaluation of Drug–Drug Interaction Evidence for Clinical Decision SupportDrug Safety, 2015
- AI TOOLS IN DECISION MAKING SUPPORT SYSTEMS: A REVIEWInternational Journal on Artificial Intelligence Tools, 2012
- Human attention and its implications for human–computer interaction2011
- Artificial intelligence in radiology: decision support systems.RadioGraphics, 1994