Evaluating Index-based Treatment Allocation in Underresourced Communities
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (27) , 27849-27857
Abstract
In many applications of AI for Social Impact (e.g., when allocating spots in support programs for underserved communities), resources are scarce and an allocation policy is needed to decide who receives a resource. Before being deployed at scale, a rigorous evaluation of an AI-powered allocation policy is vital. In this paper, we introduce the methods necessary to evaluate index-based allocation policies, which allocate a limited number of resources to those who need them the most. Such policies create dependencies between agents, rendering standard statistical tests invalid and ineffective. Addressing the arising practical and technical challenges, we describe an efficient estimator and methods for drawing valid statistical conclusions. Our extensive experiments validate our methodology in practical settings while also showcasing its statistical power. We conclude by proposing and empirically verifying extensions of our methodology that enable us to reevaluate a past randomized control trial conducted with 10000 beneficiaries for a mHealth program for pregnant women. Our new methodology allows us to draw previously invisible conclusions when comparing two different ML allocation policies.Keywords
This publication has 18 references indexed in Scilit:
- Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized ExperimentsJournal of Business & Economic Statistics, 2024
- Expanding impact of mobile health programs: SAHELI for maternal and child careAI Magazine, 2023
- Efficient and targeted COVID-19 border testing via reinforcement learningNature, 2021
- Risk-based maintenance planning of offshore wind turbine farmsReliability Engineering & System Safety, 2020
- Data-driven predictive maintenance scheduling policies for railwaysTransportation Research Part C: Emerging Technologies, 2019
- An Efficacy Study of a Ninth-Grade Early Warning Indicator InterventionJournal of Research on Educational Effectiveness, 2019
- Metalearners for estimating heterogeneous treatment effects using machine learningProceedings of the National Academy of Sciences, 2019
- Estimation and Inference of Heterogeneous Treatment Effects using Random ForestsJournal of the American Statistical Association, 2018
- Risk and treatment effect heterogeneity: re-analysis of individual participant data from 32 large clinical trialsInternational Journal of Epidemiology, 2016
- A model for allocating CDC’s HIV prevention resources in the United StatesHealth Care Management Science, 2010