Mohsen Bayati

Multiarmed bandit (MAB) algorithms are efficient approaches to reduce the opportunity cost of online experimentation and are used by companies to find the best product from periodically refreshed product catalogs. However, these algorithms face the so-called cold-start at the onset of the experiment due to a lack of knowledge of customer preferences for new products, requiring an initial data collection phase known as the burn-in period. During this period, standard MAB algorithms operate like randomized experiments, incurring large burn-in costs which scale with the large number of products. We attempt to reduce the burn-in by identifying that many products can be cast into two-sided products and then naturally model the rewards of the products with a matrix, whose rows and columns represent the two sides, respectively. Next, we design two-phase bandit algorithms that first use subsampling and low-rank matrix estimation to obtain a substantially smaller targeted set of products and then apply a Upper Confidence Bound procedure on the target products to find the best one. We theoretically show that the proposed algorithms lower costs and expedite the experiment in cases when there is limited experimentation time along with a large product set. Our analysis also provides insights into three experiment regimes of long, short, and ultra-short horizon, determined by the dimensions of the matrix. Empirical evidence from both synthetic data and a real-world data set on music streaming services validates the superior performance suggested by our theory. This paper was accepted by J. George Shanthikumar, data science. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.03394 .

Modern language model deployments must often balance competing objectives, for example, helpfulness versus harmlessness, cost versus accuracy, and reward versus safety. We introduce Conformal Arbitrage, a post hoc framework that learns a data driven threshold to mediate between a Primary model optimized for a primary objective and a more conservative Guardian which could be another model or a human domain expert aligned with a guardrail objective. The threshold is calibrated with conformal risk control, yielding finite sample, distribution free guarantees that the long run frequency of undesirable events, such as factual errors or safety violations, does not exceed a user specified quota. Because Conformal Arbitrage operates wholly at the API level, without requiring access to model logits or updating model weights, it complements weight based alignment techniques and integrates seamlessly with existing cost aware cascades. Empirically, Conformal Arbitrage traces an efficient frontier, allowing users to define an acceptable performance level for one objective while maximizing utility in another. We observe that our method outperforms, in terms of accuracy, cost matched random routing between models. These properties make Conformal Arbitrage a practical, theoretically grounded tool for trustworthy and economical deployment of large language models across a broad range of potentially competing objectives.

Background: Physician surveys provide indispensable insights into physician experience, but the question of whether responders are representative can limit confidence in conclusions. Ubiquitously collected electronic health record (EHR) use data may improve understanding of the experiences of survey nonresponders in relation to responders, providing clues regarding their well-being. Objective: The aim of the study was to identify EHR use measures corresponding with physician survey responses and examine methods to estimate population-level survey results among physicians. Methods: This longitudinal observational study was conducted from 2019 through 2020 among academic and community primary care physicians. We quantified EHR use using vendor-derived and investigator-derived measures, quantified burnout symptoms using emotional exhaustion and interpersonal disengagement subscales of the Stanford Professional Fulfillment Index, and used an ensemble of response propensity-weighted penalized linear regressions to develop a burnout symptom prediction model. Results: Among 697 surveys from 477 physicians with a response rate of 80.5% (697/866), always responders were similar to nonresponders in gender (204/340, 60% vs 38/66, 58% women; P=.78) and age (median 50, IQR 40‐60 years vs median 50, IQR 37.5‐57.5 years; P=.88) but with higher clinical workload (median 121.5, IQR 58.5‐184 vs median 34.5, IQR 0‐115 appointments; P<.001), efficiency (median 5.2, IQR 4.0-6.2 vs median 4.3, IQR 0‐5.6; P<.001), and proficiency (median 7.0, IQR 5.4‐8.5 vs median 3.1, IQR 0‐6.3; P<.001). Survey response status prediction showed an out-of-sample area under the receiver operating characteristics curve of 0.88 (95% CI 0.77-0.91). Burnout symptom prediction showed an out-of-sample area under the receiver operating characteristics curve of 0.63 (95% CI 0.57-0.70). The predicted burnout prevalence among nonresponders was 52%, higher than the observed prevalence of 28% among responders, resulting in an estimated population burnout prevalence of 31%. Conclusions: EHR use measures showed limited utility for predicting burnout symptoms but allowed discrimination between responders and nonresponders. These measures may enable qualitative interpretations of the effects of nonresponders and may inform survey response maximization efforts.

Randomized experiments are a powerful methodology for data-driven evaluation of decisions or interventions. Yet, their validity may be undermined by network interference. This occurs when the treatment of one unit impacts not only its outcome but also that of connected units, biasing traditional treatment effect estimations. Our study introduces a framework to accommodate complex and unknown network interference, moving beyond specialized models in the existing literature. Our framework, termed causal message-passing, is grounded in high-dimensional approximate message-passing methodology. It is tailored for multiperiod experiments and is particularly effective in settings with many units and prevalent network interference. The framework models causal effects as a dynamic process where a treated unit’s impact propagates through the network via neighboring units until equilibrium is reached. This approach allows us to approximate the dynamics of potential outcomes over time, enabling the extraction of valuable information before treatment effects reach equilibrium. Utilizing causal message-passing, we introduce a practical algorithm to estimate the total treatment effect, defined as the impact observed when all units are treated compared to the scenario where no unit receives treatment. We demonstrate the effectiveness of this approach across five numerical scenarios, each characterized by a distinct interference structure.

In this paper, we study the design and analysis of experiments conducted on a set of units over multiple time periods in which the starting time of the treatment may vary by unit. The design problem involves selecting an initial treatment time for each unit in order to most precisely estimate both the instantaneous and cumulative effects of the treatment. We first consider nonadaptive experiments, in which all treatment assignment decisions are made prior to the start of the experiment. For this case, we show that the optimization problem is generally NP-hard, and we propose a near-optimal solution. Under this solution, the fraction entering treatment each period is initially low, then high, and finally low again. Next, we study an adaptive experimental design problem, in which both the decision to continue the experiment and treatment assignment decisions are updated after each period’s data are collected. For the adaptive case, we propose a new algorithm, the precision-guided adaptive experiment algorithm, which addresses the challenges at both the design stage and the stage of estimating treatment effects, ensuring valid post-experiment inference, accounting for the adaptive nature of the design. Using realistic settings, we demonstrate that our proposed solutions can reduce the opportunity cost of the experiments by more than 50%, compared with static design benchmarks. This paper was accepted by George Shanthikumar, data science. Funding: S. Athey and G. Imbens were supported by the Office of Naval Research [Grant N00014-19-1-2468]. M. Bayati was supported by the National Science Foundation [Grant CMMI 1554140]. Supplemental Material: The e-companion and data are available at https://doi.org/10.1287/mnsc.2023.4928 .

BACKGROUND Physician surveys provide indispensable insights into physician experience, but the question of whether responders are representative can limit confidence in conclusions. Ubiquitously-collected electronic health record (EHR) usage data may improve understanding of the experiences of survey non-responders in relation to responders, providing clues regarding their well-being. OBJECTIVE To identify EHR usage measures corresponding with physician survey response and examine methods to estimate population-level survey results among physicians. METHODS Longitudinal observational study from 2019 through 2020, among academic and community primary care physicians. We quantified EHR usage using vendor-derived and investigator-derived measures, quantified burnout symptoms using emotional exhaustion and interpersonal disengagement subscales of the Stanford Professional Fulfillment Index, and used an ensemble of response propensity-weighted penalized linear regressions to develop a burnout symptom prediction model. RESULTS Among 697 surveys from 477 physicians, always-responders were similar to non-responders in gender [204 (60%) vs 38 (58%) women, P=.22] and age [50 (IQR 40-60) vs. 50 (IQR 37.5-57.5) years, P=.43], but with higher clinical workload [148.5 (IQR 71-232) vs. 39 (IQR 0-116) appointments, P<0.001], efficiency [5.1 (IQR 3.9-6.1) vs. 4.5 (IQR 0-5.7), P<0.001], and proficiency [7.0 (IQR 5.5-8.5) vs. 3.3 (0-6.3), P<0.001]. Survey response status prediction showed an out-of-sample AUROC 0.88 (95%CI 0.77-0.91). Burnout symptom prediction showed an out-of-sample AUROC 0.63 (95%CI 0.57-0.70). Predicted burnout prevalence among non-responders was 52%, higher than the observed prevalence of 28% among responders, resulting in an estimated population burnout prevalence of 31%. CONCLUSIONS EHR usage measures showed limited utility for predicting burnout symptoms, but allowed discrimination between responders and non-responders. These measures may enable qualitative interpretations of the effects of non-responders and may inform survey response maximization efforts.

In this paper, we study the trace regression when a matrix of parameters B* is estimated via the convex relaxation of a rank-regularized regression or via regularized non-convex optimization. It is known that these estimators satisfy near-optimal error bounds under assumptions on the rank, coherence, and spikiness of B*. We start by introducing a general notion of spikiness for B* that provides a generic recipe to prove the restricted strong convexity of the sampling operator of the trace regression and obtain near-optimal and non-asymptotic error bounds for the estimation error. Similar to the existing literature, these results require the regularization parameter to be above a certain theory-inspired threshold that depends on observation noise that may be unknown in practice. Next, we extend the error bounds to cases where the regularization parameter is chosen via cross-validation. This result is significant in that existing theoretical results on cross-validated estimators (Kale et al., 2011; Kumar et al., 2013; Abou-Moustafa and Szepesvari, 2017) do not apply to our setting since the estimators we study are not known to satisfy their required notion of stability. Finally, using simulations on synthetic and real data, we show that the cross-validated estimator selects a near-optimal penalty parameter and outperforms the theory-inspired approach of selecting the parameter.

A General Elliptical Potential Lemma In sequential learning and decision-making problems, the elliptical potential lemma is a key technique to quantify the decrease in the uncertainty of the model as more observations are obtained. However, it requires the observation noise and prior distribution of the unknown parameters to be Gaussian. In “The Elliptical Potential Lemma for General Distributions with an Application to Linear Thompson Sampling,” N. Hamidi and M. Bayati introduce a general version of the elliptical potential lemma that relaxes the Gaussian assumption. They also apply their general lemma to prove a minimax optimal Bayesian regret bound for the well-known Thompson sampling algorithm in stochastic linear bandits with changing action sets where prior and noise distributions are general.