Art B. Owen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Art B. Owen has a total of 84 co-authors. Here are the top co-authors by number of shared publications:
- Edgar Dobriban
University of Pennsylvania
3 shared publications Stuart K. Kim
Department of Developmental Biology Department of Genetics Stanford University 94305 Stanford California U.S.A
3 shared publications- Dan M. Kluger
Institute for Data Systems, and Society Massachusetts Institute of Technology
2 shared publications - Dan M. Kluger
Department of Statistics Stanford University a
2 shared publications - Jingshu Wang
Department of Statistics, University of Chicago
2 shared publications - Masayoshi Mase
Research & Development Group Hitachi, Ltd
2 shared publications
Latest Publications
- Open Access
On the optimization of discrepancy measures
Points in the unit cube with low discrepancy can be constructed using algebra or, more recently, by direct computational optimization of a criterion. The usual $L_\infty$ star discrepancy is a poor criterion for this because it is computationally expensive and lacks differentiability. Its usual replacement, the $L_2$ star discrepancy, is smooth but exhibits other pathologies shown by J. Matoušek. In an attempt to address these problems, we introduce the \textit{average squared discrepancy} which averages over $2^d$ versions of the $L_2$ star discrepancy anchored in the different vertices of $[0,1]^d$. Not only can this criterion be computed in $O(dn^2)$ time, like the $L_2$ star discrepancy, but also we show that it is equivalent to a weighted symmetric $L_2$ criterion of Hickernell's by a constant factor. We compare this criterion with a wide range of traditional discrepancy measures, and show that only the average squared discrepancy avoids the problems raised by Matoušek. Furthermore, we present a comprehensive numerical study showing in particular that optimizing for the average squared discrepancy leads to strong performance for the $L_2$ star discrepancy, whereas the converse does not hold.Open AccessOn the optimization of discrepancy measures
Points in the unit cube with low discrepancy can be constructed using algebra or, more recently, by direct computational optimization of a criterion. The usual $L_\infty$ star discrepancy is a poor criterion for this because it is computationally expensive and lacks differentiability. Its usual replacement, the $L_2$ star discrepancy, is smooth but exhibits other pathologies shown by J. Matoušek. In an attempt to address these problems, we introduce the \textit{average squared discrepancy} which averages over $2^d$ versions of the $L_2$ star discrepancy anchored in the different vertices of $[0,1]^d$. Not only can this criterion be computed in $O(dn^2)$ time, like the $L_2$ star discrepancy, but also we show that it is equivalent to a weighted symmetric $L_2$ criterion of Hickernell's by a constant factor. We compare this criterion with a wide range of traditional discrepancy measures, and show that only the average squared discrepancy avoids the problems raised by Matoušek. Furthermore, we present a comprehensive numerical study showing in particular that optimizing for the average squared discrepancy leads to strong performance for the $L_2$ star discrepancy, whereas the converse does not hold. - Open Access
Biases in estimates of air pollution impacts: the role of omitted variables and measurement errors
Observational studies often use linear regression to assess the effect of ambient air pollution on outcomes of interest, such as human health indicators or crop yields. Yet pollution datasets are typically noisy and include only a subset of the potentially relevant pollutants, giving rise to both measurement error bias (MEB) and omitted variable bias (OVB). While it is well understood that these biases exist, less is understood about whether these biases tend to be positive or negative, even though it is sometimes falsely claimed that measurement error simply biases regression coefficient estimates towards zero. In this paper, we study the direction of these biases under the realistic assumptions that the concentrations of different types of air pollutants are positively correlated with each other and that each type of pollutant has a nonpositive association with the outcome variable. We demonstrate both theoretically and using simulations that under these two assumptions, the OVB will typically be negative and that more often than not the MEB for null pollutants or for pollutants that are perfectly measured will be negative. We also use a crop yield and air pollution dataset to show that these biases tend to be negative in the setting of our motivating application. We do this by introducing a validation scheme that does not require knowing the true coefficients. While this paper is motivated by studies assessing the effect of air pollutants on crop yields, the findings are also relevant to regression-based studies assessing the effect of air pollutants on human health outcomes. The validation scheme can also be used to empirically study OVB or MEB in other contexts.
Open AccessBiases in estimates of air pollution impacts: the role of omitted variables and measurement errors
Observational studies often use linear regression to assess the effect of ambient air pollution on outcomes of interest, such as human health indicators or crop yields. Yet pollution datasets are typically noisy and include only a subset of the potentially relevant pollutants, giving rise to both measurement error bias (MEB) and omitted variable bias (OVB). While it is well understood that these biases exist, less is understood about whether these biases tend to be positive or negative, even though it is sometimes falsely claimed that measurement error simply biases regression coefficient estimates towards zero. In this paper, we study the direction of these biases under the realistic assumptions that the concentrations of different types of air pollutants are positively correlated with each other and that each type of pollutant has a nonpositive association with the outcome variable. We demonstrate both theoretically and using simulations that under these two assumptions, the OVB will typically be negative and that more often than not the MEB for null pollutants or for pollutants that are perfectly measured will be negative. We also use a crop yield and air pollution dataset to show that these biases tend to be negative in the setting of our motivating application. We do this by introducing a validation scheme that does not require knowing the true coefficients. While this paper is motivated by studies assessing the effect of air pollutants on crop yields, the findings are also relevant to regression-based studies assessing the effect of air pollutants on human health outcomes. The validation scheme can also be used to empirically study OVB or MEB in other contexts.
- 1 February 2024
- Published by Bernoulli Society for Mathematical Statistics and Probability in Bernoulli
- Vol. 30 (1) , 743-769
A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
The Benjamini-Hochberg (BH) procedure remains widely popular despite having limited theoretical guarantees in the commonly encountered scenario of correlated test statistics. Of particular concern is the possibility that the method could exhibit bursty behavior, meaning that it might typically yield no false discoveries while occasionally yielding both a large number of false discoveries and a false discovery proportion (FDP) that far exceeds its own well controlled mean. In this paper, we investigate which test statistic correlation structures lead to bursty behavior and which ones lead to well controlled FDPs. To this end, we develop a central limit theorem for the FDP in a multiple testing setup where the test statistic correlations can be either short-range or long-range as well as either weak or strong. The theorem and our simulations from a data-driven factor model suggest that the BH procedure exhibits severe burstiness when the test statistics have many strong, long-range correlations, but does not otherwise.
- 1 February 2024
- Published by Bernoulli Society for Mathematical Statistics and Probability in Bernoulli
- Vol. 30 (1) , 743-769
A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
The Benjamini-Hochberg (BH) procedure remains widely popular despite having limited theoretical guarantees in the commonly encountered scenario of correlated test statistics. Of particular concern is the possibility that the method could exhibit bursty behavior, meaning that it might typically yield no false discoveries while occasionally yielding both a large number of false discoveries and a false discovery proportion (FDP) that far exceeds its own well controlled mean. In this paper, we investigate which test statistic correlation structures lead to bursty behavior and which ones lead to well controlled FDPs. To this end, we develop a central limit theorem for the FDP in a multiple testing setup where the test statistic correlations can be either short-range or long-range as well as either weak or strong. The theorem and our simulations from a data-driven factor model suggest that the BH procedure exhibits severe burstiness when the test statistics have many strong, long-range correlations, but does not otherwise.
- Open Access
- 1 January 2024
- Published by Institute of Mathematical Statistics in Electronic Journal of Statistics
- Vol. 18 (2) , 4612-4639
Multivariate tie-breaker designs
In a tie-breaker design (TBD), subjects with high values of a running variable are given some (usually desirable) treatment, subjects with low values are not, and subjects in the middle are randomized. TBDs are intermediate between regression discontinuity designs (RDDs) and randomized controlled trials (RCTs). TBDs allow a tradeoff between the resource allocation efficiency of an RDD and the statistical efficiency of an RCT. We study a model where the expected response is one multivariate regression for treated subjects and another for control subjects. We propose a prospective D-optimality, analogous to Bayesian optimal design, to understand design tradeoffs without reference to a specific data set. For given covariates, we show how to use convex optimization to choose treatment probabilities that optimize this criterion. We can incorporate a variety of constraints motivated by economic and ethical considerations. In our model, D-optimality for the treatment effect coincides with D-optimality for the whole regression, and, without constraints, an RCT is globally optimal. We show that a monotonicity constraint favoring more deserving subjects induces sparsity in the number of distinct treatment probabilities. We apply the convex optimization solution to a semi-synthetic example involving triage data from the MIMIC-IV-ED database.
- 1 January 2024
- Published by Institute of Mathematical Statistics in Electronic Journal of Statistics
- Vol. 18 (2) , 4612-4639
Open AccessMultivariate tie-breaker designs
In a tie-breaker design (TBD), subjects with high values of a running variable are given some (usually desirable) treatment, subjects with low values are not, and subjects in the middle are randomized. TBDs are intermediate between regression discontinuity designs (RDDs) and randomized controlled trials (RCTs). TBDs allow a tradeoff between the resource allocation efficiency of an RDD and the statistical efficiency of an RCT. We study a model where the expected response is one multivariate regression for treated subjects and another for control subjects. We propose a prospective D-optimality, analogous to Bayesian optimal design, to understand design tradeoffs without reference to a specific data set. For given covariates, we show how to use convex optimization to choose treatment probabilities that optimize this criterion. We can incorporate a variety of constraints motivated by economic and ethical considerations. In our model, D-optimality for the treatment effect coincides with D-optimality for the whole regression, and, without constraints, an RCT is globally optimal. We show that a monotonicity constraint favoring more deserving subjects induces sparsity in the number of distinct treatment probabilities. We apply the convex optimization solution to a semi-synthetic example involving triage data from the MIMIC-IV-ED database.
- Open Access
- 1 January 2024
- Published by Walter de Gruyter GmbH in Journal of Causal Inference
- Vol. 12 (1)
Double machine learning and design in batch adaptive experiments
We consider an experiment with at least two stages or batches and subjects per batch. First, we propose a semiparametric treatment effect estimator that efficiently pools information across the batches, and we show that it asymptotically dominates alternatives that aggregate single batch estimates. Then, we consider the design problem of learning propensity scores for assigning treatment in the later batches of the experiment to maximize the asymptotic precision of this estimator. For two common causal estimands, we estimate this precision using observations from previous batches, and then solve a finite-dimensional concave maximization problem to adaptively learn flexible propensity scores that converge to suitably defined optima in each batch at rate . By extending the framework of double machine learning, we show this rate suffices for our pooled estimator to attain the targeted precision after each batch, as long as nuisance function estimates converge at rate . These relatively weak rate requirements enable the investigator to avoid the common practice of discretizing the covariate space for design and estimation in batch adaptive experiments while maintaining the advantages of pooling. Our numerical study shows that such discretization often leads to substantial asymptotic and finite sample precision losses outweighing any gains from design.- 1 January 2024
- Published by Walter de Gruyter GmbH in Journal of Causal Inference
- Vol. 12 (1)
Open AccessDouble machine learning and design in batch adaptive experiments
We consider an experiment with at least two stages or batches and subjects per batch. First, we propose a semiparametric treatment effect estimator that efficiently pools information across the batches, and we show that it asymptotically dominates alternatives that aggregate single batch estimates. Then, we consider the design problem of learning propensity scores for assigning treatment in the later batches of the experiment to maximize the asymptotic precision of this estimator. For two common causal estimands, we estimate this precision using observations from previous batches, and then solve a finite-dimensional concave maximization problem to adaptively learn flexible propensity scores that converge to suitably defined optima in each batch at rate . By extending the framework of double machine learning, we show this rate suffices for our pooled estimator to attain the targeted precision after each batch, as long as nuisance function estimates converge at rate . These relatively weak rate requirements enable the investigator to avoid the common practice of discretizing the covariate space for design and estimation in batch adaptive experiments while maintaining the advantages of pooling. Our numerical study shows that such discretization often leads to substantial asymptotic and finite sample precision losses outweighing any gains from design. - Open Access
- 16 October 2023
- Published by Oxford University Press (OUP) in Journal of the Royal Statistical Society Series B: Statistical Methodology
- Vol. 86 (1) , 47-48
Art Owen’s contribution to the Discussion of ‘Estimating means of bounded random variables by betting’ by Waudby-Smith and Ramdas
Art B Owen; Art Owen's contribution to the Discussion of “Estimating means of bounded random variables by betting” by Ian Waudby-Smith and Aaditya Ramdas, Journ
- 16 October 2023
- Published by Oxford University Press (OUP) in Journal of the Royal Statistical Society Series B: Statistical Methodology
- Vol. 86 (1) , 47-48
Open AccessArt Owen’s contribution to the Discussion of ‘Estimating means of bounded random variables by betting’ by Waudby-Smith and Ramdas
Art B Owen; Art Owen's contribution to the Discussion of “Estimating means of bounded random variables by betting” by Ian Waudby-Smith and Aaditya Ramdas, Journ
- Open Access
Deletion and Insertion Tests in Regression Models
A basic task in explainable AI (XAI) is to identify the most important features behind a prediction made by a black box function $f$. The insertion and deletion tests of Petsiuk et al. (2018) can be used to judge the quality of algorithms that rank pixels from most to least important for a classification. Motivated by regression problems we establish a formula for their area under the curve (AUC) criteria in terms of certain main effects and interactions in an anchored decomposition of $f$. We find an expression for the expected value of the AUC under a random ordering of inputs to $f$ and propose an alternative area above a straight line for the regression setting. We use this criterion to compare feature importances computed by integrated gradients (IG) to those computed by Kernel SHAP (KS) as well as LIME, DeepLIFT, vanilla gradient and input$\times$gradient methods. KS has the best overall performance in two datasets we consider but it is very expensive to compute. We find that IG is nearly as good as KS while being much faster. Our comparison problems include some binary inputs that pose a challenge to IG because it must use values between the possible variable levels and so we consider ways to handle binary variables in IG. We show that sorting variables by their Shapley value does not necessarily give the optimal ordering for an insertion-deletion test. It will however do that for monotone functions of additive models, such as logistic regression.Open AccessDeletion and Insertion Tests in Regression Models
A basic task in explainable AI (XAI) is to identify the most important features behind a prediction made by a black box function $f$. The insertion and deletion tests of Petsiuk et al. (2018) can be used to judge the quality of algorithms that rank pixels from most to least important for a classification. Motivated by regression problems we establish a formula for their area under the curve (AUC) criteria in terms of certain main effects and interactions in an anchored decomposition of $f$. We find an expression for the expected value of the AUC under a random ordering of inputs to $f$ and propose an alternative area above a straight line for the regression setting. We use this criterion to compare feature importances computed by integrated gradients (IG) to those computed by Kernel SHAP (KS) as well as LIME, DeepLIFT, vanilla gradient and input$\times$gradient methods. KS has the best overall performance in two datasets we consider but it is very expensive to compute. We find that IG is nearly as good as KS while being much faster. Our comparison problems include some binary inputs that pose a challenge to IG because it must use values between the possible variable levels and so we consider ways to handle binary variables in IG. We show that sorting variables by their Shapley value does not necessarily give the optimal ordering for an insertion-deletion test. It will however do that for monotone functions of additive models, such as logistic regression. - Open Access
A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
The Benjamini-Hochberg (BH) procedure remains widely popular despite having limited theoretical guarantees in the commonly encountered scenario of correlated test statistics. Of particular concern is the possibility that the method could exhibit bursty behavior, meaning that it might typically yield no false discoveries while occasionally yielding both a large number of false discoveries and a false discovery proportion (FDP) that far exceeds its own well controlled mean. In this paper, we investigate which test statistic correlation structures lead to bursty behavior and which ones lead to well controlled FDPs. To this end, we develop a central limit theorem for the FDP in a multiple testing setup where the test statistic correlations can be either short-range or long-range as well as either weak or strong. The theorem and our simulations from a data-driven factor model suggest that the BH procedure exhibits severe burstiness when the test statistics have many strong, long-range correlations, but does not otherwise.
Open AccessA central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
The Benjamini-Hochberg (BH) procedure remains widely popular despite having limited theoretical guarantees in the commonly encountered scenario of correlated test statistics. Of particular concern is the possibility that the method could exhibit bursty behavior, meaning that it might typically yield no false discoveries while occasionally yielding both a large number of false discoveries and a false discovery proportion (FDP) that far exceeds its own well controlled mean. In this paper, we investigate which test statistic correlation structures lead to bursty behavior and which ones lead to well controlled FDPs. To this end, we develop a central limit theorem for the FDP in a multiple testing setup where the test statistic correlations can be either short-range or long-range as well as either weak or strong. The theorem and our simulations from a data-driven factor model suggest that the BH procedure exhibits severe burstiness when the test statistics have many strong, long-range correlations, but does not otherwise.
- Open Access
- 23 April 2023
- Published by The Open Journal in Journal of Open Source Software
- Vol. 8 (84) , 5309
Quasi-Monte Carlo Methods in Python
NumPy random number generators and SciPy distributions are widely used to generate random numbers. However, challenges might arise when sampling in high dimensions. Quasi-Monte Carlo (QMC) methods provide an answer to these problems but are arguably hard to use. Thanks to new developments in SciPy, a new submodule was introduced in version 1.7.0 making state-of-the-art QMC methods available: scipy.stats.qmc.
- 23 April 2023
- Published by The Open Journal in Journal of Open Source Software
- Vol. 8 (84) , 5309
Open AccessQuasi-Monte Carlo Methods in Python
NumPy random number generators and SciPy distributions are widely used to generate random numbers. However, challenges might arise when sampling in high dimensions. Quasi-Monte Carlo (QMC) methods provide an answer to these problems but are arguably hard to use. Thanks to new developments in SciPy, a new submodule was introduced in version 1.7.0 making state-of-the-art QMC methods available: scipy.stats.qmc.
- Open Access
Model free variable importance for high dimensional data
A model-agnostic variable importance method can be used with arbitrary prediction functions. Here we present some model-free methods that do not require access to the prediction function. This is useful when that function is proprietary and not available, or just extremely expensive. It is also useful when studying residuals from a model. The cohort Shapley (CS) method is model-free but has exponential cost in the dimension of the input space. A supervised on-manifold Shapley method from Frye et al. (2020) is also model free but requires as input a second black box model that has to be trained for the Shapley value problem. We introduce an integrated gradient (IG) version of cohort Shapley, called IGCS, with cost $\mathcal{O}(nd)$. We show that over the vast majority of the relevant unit cube that the IGCS value function is close to a multilinear function for which IGCS matches CS. Another benefit of IGCS is that is allows IG methods to be used with binary predictors. We use some area between curves (ABC) measures to quantify the performance of IGCS. On a problem from high energy physics we verify that IGCS has nearly the same ABCs as CS does. We also use it on a problem from computational chemistry in 1024 variables. We see there that IGCS attains much higher ABCs than we get from Monte Carlo sampling. The code is publicly available at https://github.com/cohortshapley/cohortintgradOpen AccessModel free variable importance for high dimensional data
A model-agnostic variable importance method can be used with arbitrary prediction functions. Here we present some model-free methods that do not require access to the prediction function. This is useful when that function is proprietary and not available, or just extremely expensive. It is also useful when studying residuals from a model. The cohort Shapley (CS) method is model-free but has exponential cost in the dimension of the input space. A supervised on-manifold Shapley method from Frye et al. (2020) is also model free but requires as input a second black box model that has to be trained for the Shapley value problem. We introduce an integrated gradient (IG) version of cohort Shapley, called IGCS, with cost $\mathcal{O}(nd)$. We show that over the vast majority of the relevant unit cube that the IGCS value function is close to a multilinear function for which IGCS matches CS. Another benefit of IGCS is that is allows IG methods to be used with binary predictors. We use some area between curves (ABC) measures to quantify the performance of IGCS. On a problem from high energy physics we verify that IGCS has nearly the same ABCs as CS does. We also use it on a problem from computational chemistry in 1024 variables. We see there that IGCS attains much higher ABCs than we get from Monte Carlo sampling. The code is publicly available at https://github.com/cohortshapley/cohortintgrad