Susan Athey

Effective information sharing is critical for the success of organizations and governments. Because information that is easy to access is more likely to be adopted, leaders often minimize friction in information delivery. However, one type of friction may increase engagement: piquing curiosity by posing relevant questions prior to sharing information. To test this, we shared identical information about COVID-19 in either question-and-answer format or via direct statements across two preregistered field experiments in Ghana and Michigan (total n = 49,395). Q&A-style communication increased information seeking about directly related topics (e.g., how to wear a mask properly) by 1.0 percentage point (216%) in Ghana and by 1.1 percentage points (19%) in Michigan (p’s < 0.001) and increased self-reported behavior change by 1.3 percentage points (4%) in Michigan (p = 0.002). However, sharing information in Q&A format did not increase interest in general COVID-19 information in either setting, suggesting that the impact of Q&A-style messaging on information seeking may be issue specific. In Michigan, both Q&A-style and direct statement messaging produced less information seeking than sending no informational messages, likely because of differential attrition: the more texts participants received, the more likely they were to opt out of receiving messages, which made it impossible for them to seek more information via text. In a follow-up implementation experiment with social media ads (a messaging strategy without attrition challenges), Q&A-style ads generated 9%–11% more unique clicks to the CDC website per dollar spent than ads that directly stated information about vaccines (p < 0.001). We speculate that Q&A-style information delivery may stimulate curiosity, driving its benefits. This paper was accepted by Marie Claire Villeval, behavioral economics and decision analysis. Funding: The authors thank the National Science Foundation [RAPID Grant 2033321], the Bill and Melinda Gates Foundation, Northwestern University’s Global Poverty Research Lab, Stanford University’s Golub Capital Social Impact Lab, Harvard Business School, the University of Pennsylvania, the AKO Foundation, John Alexander, Mark J. Leder, and Warren G. Lichtenstein for funding support. This work was also supported by Grand Challenges in Global Health. Supplemental Material: The supplementary materials and data files are available at https://doi.org/10.1287/mnsc.2024.04405 .

The rise of foundation models marks a paradigm shift in machine learning: instead of training specialized models from scratch, foundation models are trained on massive datasets before being adjusted or fine-tuned to make predictions on smaller datasets. Initially developed for text, foundation models have also excelled at making predictions about social science data. However, while many estimation problems in the social sciences use prediction as an intermediate step, they ultimately require different criteria for success. In this paper, we develop methods for fine-tuning foundation models to perform these estimation problems. We first characterize an omitted variable bias that can arise when a foundation model is fine-tuned in the standard way: to minimize predictive error. We then provide a set of conditions for fine-tuning under which estimates derived from a foundation model are n -consistent. Based on this theory, we develop fine-tuning algorithms that empirically mitigate this omitted variable bias. To demonstrate our ideas, we study gender wage gap estimation. Classical methods for estimating the adjusted wage gap employ simple predictive models of wages, which can induce omitted variable bias because they condition on coarse summaries of career history. Instead, we use a custom-built foundation model, capturing a richer representation of career history. Using data from the Panel Study of Income Dynamics, we find that career history explains more of the gender wage gap than standard econometric models can measure, and we identify elements of career history that are omitted by standard models but are important for explaining the gap.

In this paper, we study the design and analysis of experiments conducted on a set of units over multiple time periods in which the starting time of the treatment may vary by unit. The design problem involves selecting an initial treatment time for each unit in order to most precisely estimate both the instantaneous and cumulative effects of the treatment. We first consider nonadaptive experiments, in which all treatment assignment decisions are made prior to the start of the experiment. For this case, we show that the optimization problem is generally NP-hard, and we propose a near-optimal solution. Under this solution, the fraction entering treatment each period is initially low, then high, and finally low again. Next, we study an adaptive experimental design problem, in which both the decision to continue the experiment and treatment assignment decisions are updated after each period’s data are collected. For the adaptive case, we propose a new algorithm, the precision-guided adaptive experiment algorithm, which addresses the challenges at both the design stage and the stage of estimating treatment effects, ensuring valid post-experiment inference, accounting for the adaptive nature of the design. Using realistic settings, we demonstrate that our proposed solutions can reduce the opportunity cost of the experiments by more than 50%, compared with static design benchmarks. This paper was accepted by George Shanthikumar, data science. Funding: S. Athey and G. Imbens were supported by the Office of Naval Research [Grant N00014-19-1-2468]. M. Bayati was supported by the National Science Foundation [Grant CMMI 1554140]. Supplemental Material: The e-companion and data are available at https://doi.org/10.1287/mnsc.2023.4928 .

In a wide variety of applications, including healthcare, bidding in first price auctions, digital recommendations, and online education, it can be beneficial to learn a policy that assigns treatments to individuals based on their characteristics. The growing policy-learning literature focuses on settings in which policies are learned from historical data in which the treatment assignment rule is fixed throughout the data-collection period. However, adaptive data collection is becoming more common in practice from two primary sources: (1) data collected from adaptive experiments that are designed to improve inferential efficiency and (2) data collected from production systems that progressively evolve an operational policy to improve performance over time (e.g., contextual bandits). Yet adaptivity complicates the problem of learning an optimal policy ex post for two reasons: first, samples are dependent and, second, an adaptive assignment rule may not assign each treatment to each type of individual sufficiently often. In this paper, we address these challenges. We propose an algorithm based on generalized augmented inverse propensity weighted (AIPW) estimators, which nonuniformly reweight the elements of a standard AIPW estimator to control worst case estimation variance. We establish a finite-sample regret upper bound for our algorithm and complement it with a regret lower bound that quantifies the fundamental difficulty of policy learning with adaptive data. When equipped with the best weighting scheme, our algorithm achieves minimax rate-optimal regret guarantees even with diminishing exploration. Finally, we demonstrate our algorithm’s effectiveness using both synthetic data and public benchmark data sets. This paper was accepted by Hamid Nazerzadeh, data science. Funding: This work is supported by the National Science Foundation [Grant CCF-2106508]. R. Zhan was supported by Golub Capital and the Michael Yao and Sara Keying Dai AI and Digital Technology Fund. Z. Ren was supported by the Office of Naval Research [Grant N00014-20-1-2337]. S. Athey was supported by the Office of Naval Research [Grant N00014-19-1-2468]. Z. Zhou is generously supported by the New York University’s 2022–2023 Center for Global Economy and Business faculty research grant and the Digital Twin research grant from Bain & Company. Supplemental Material: The data files are available at https://doi.org/10.1287/mnsc.2023.4921 .

Estimating the long-term effects of treatments is of interest in many fields. A common challenge in estimating such treatment effects is that long-term outcomes are unobserved in the time frame needed to make policy decisions. One approach to overcome this missing data problem is to analyze treatments effects on an intermediate outcome, often called a statistical surrogate, if it satisfies the condition that treatment and outcome are independent conditional on the statistical surrogate. The validity of the surrogacy condition is often controversial. Here we exploit that fact that in modern datasets, researchers often observe a large number, possibly hundreds or thousands, of intermediate outcomes, thought to lie on or close to the causal chain between the treatment and the long-term outcome of interest. Even if none of the individual proxies satisfies the statistical surrogacy criterion by itself, using multiple proxies can be useful in causal inference. We focus primarily on a setting with two samples, an experimental sample containing data about the treatment indicator and the surrogates and an observational sample containing information about the surrogates and the primary outcome. We state assumptions under which the average treatment effect be identified and estimated with a high-dimensional vector of proxies that collectively satisfy the surrogacy assumption, and derive the bias from violations of the surrogacy assumption, and show that even if the primary outcome is also observed in the experimental sample, there is still information to be gained from using surrogates.

In many applications, e.g. in healthcare and e-commerce, the goal of a contextual bandit may be to learn an optimal treatment assignment policy at the end of the experiment. That is, to minimize simple regret. However, this objective remains understudied. We propose a new family of computationally efficient bandit algorithms for the stochastic contextual bandit setting, where a tuning parameter determines the weight placed on cumulative regret minimization (where we establish near-optimal minimax guarantees) versus simple regret minimization (where we establish state-of-the-art guarantees). Our algorithms work with any function class, are robust to model misspecification, and can be used in continuous arm settings. This flexibility comes from constructing and relying on "conformal arm sets" (CASs). CASs provide a set of arms for every context, encompassing the context-specific optimal arm with a certain probability across the context distribution. Our positive results on simple and cumulative regret guarantees are contrasted with a negative result, which shows that no algorithm can achieve instance-dependent simple regret guarantees while simultaneously achieving minimax optimal cumulative regret guarantees.

This paper analyzes a randomized controlled trial of a personalized digital counseling intervention addressing informational constraints and choice architecture, cross-randomized with discounts for long-acting reversible contraceptives (LARCs), such as intrauterine devices (IUDs). The counseling intervention encourages shared decision-making (SDM) using a tablet-based app, which provides a tailored ranking of modern methods to each client according to their elicited needs and preferences. Take-up of LARCs in the status quo regime at full price was 11%, which increased to 28% with discounts. SDM roughly tripled the share of clients adopting a LARC at full price to 35%, and discounts had no incremental impact in this group. Neither intervention affected the take-up of short-acting methods, such as the pill. Consistent with theoretical models of consumer search, SDM clients discussed more methods in depth, which led to higher adoption rates for second- or lower-ranked LARCs. Our findings suggest that low-cost individualized recommendations can potentially be as effective in increasing unfamiliar technology adoption as providing large subsidies.

We develop new semiparametric methods for estimating treatment effects. We focus on settings where the outcome distributions may be thick tailed, where treatment effects may be small, where sample sizes are large and where assignment is completely random. This setting is of particular interest in recent online experimentation. We propose using parametric models for the treatment effects, leading to semiparametric models for the outcome distributions. We derive the semiparametric efficiency bound for the treatment effects for this setting, and propose efficient estimators. In the leading case with constant quantile treatment effects one of the proposed efficient estimators has an interesting interpretation as a weighted average of quantile treatment effects, with the weights proportional to minus the second derivative of the log of the density of the potential outcomes. Our analysis also suggests an extension of Huber's model and trimmed mean to include asymmetry.