HLHenry Lam

Henry Lam

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Henry Lam has a total of 90 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Orthounimodal Distributionally Robust Optimization: Representation, Computation, and Multivariate Extreme Event Applications

    by Henry Lam, Zhenyuan Liu, Xinyu Zhang

    This paper studies a basic notion of distributional shape known as orthounimodality (OU) and its use in shape-constrained distributionally robust optimization (DRO). As a key motivation, we argue how such type of DRO is well suited to tackle multivariate extreme event estimation by giving statistically valid confidence bounds on target extremal probabilities. In particular, we explain how our approach can alleviate the bias–variance trade-off and other technical complications in conventional extreme value theory, while gaining interpretability and robustness compared with existing distributional shape notions used in the DRO literature. We characterize the extreme points of the OU distribution class in terms of what we call OU sets and build a corresponding Choquet representation, which subsequently allows us to reduce DRO under OU constraints into moment problems over infinite-dimensional random variables. We then develop, in the bivariate setting, a geometric approach to reduce such moment problems into finite dimension via a specially constructed variational problem designed to eliminate suboptimal solutions. Numerical results illustrate how our approach gives rise to valid and competitive confidence bounds for extremal probabilities. Funding: This work was supported by the National Science Foundation [Grants CMMI-1834710 and IIS-1849280]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2023.0143 .

  • Certifiable Deep Importance Sampling for Rare-Event Simulation of Black Box Systems

    by Mansur Arief, Yuanlu Bai, Wenhao Ding, Shengyi He, Zhiyuan Huang, Henry Lam, Ding Zhao

    AI-Enhanced Monte Carlo Methodology Achieves More Reliable Testing and Efficiency Certificate for Safety-Critical Systems As artificial intelligence (AI) increasingly drives human-interacting intelligent physical systems such as self-driving vehicles, ensuring their safety against rare catastrophic events has also become a critical challenge. Traditional simulation-based testing methods are powerful in quantifying risks before real deployment. However, for black box systems that generically arise in AI-embedded applications, these methods can consistently underestimate the probabilities of rare failures and, dangerously, without diagnostically detectable signals. To address this perilous gap, the paper "Certifiable Deep Importance Sampling for Rare-Event Simulation of Black Box Systems," by Arief, Bai, Ding, He, Huang, Lam, and Zhao, introduces a framework based on an integration of neural networks into importance sampling, a variance reduction technique that has been found useful in rare-event simulation. This framework, which is called deep probabilistic accelerated evaluation (Deep-PrAE), uses deep neural network classifiers to learn rare-event geometries, which then calibrates importance samplers to estimate rare events with certifiable statistical guarantees. Moreover, these guarantees are designed to go beyond the conventional certifiable notions in rare-event simulation to effectively balance the applicability to black box settings with the required sampling efficiency gain that pertains to rare-event computation. These findings demonstrate a vital step forward in the reliable and certifiable deployment of complex autonomous technologies.

  • Higher-Order Expansion and Bartlett Correctability of Distributionally Robust Optimization

    by Shengyi He, Henry Lam

    Distributionally robust optimization (DRO) is a worst-case framework for stochastic optimization under uncertainty that has drawn fast-growing studies in recent years. When the underlying probability distribution is unknown and observed from data, DRO suggests computing the worst-case distribution within a so-called uncertainty set that captures the involved statistical uncertainty. In particular, DRO with uncertainty set constructed as a statistical divergence neighborhood ball has been shown to provide a tool for constructing valid confidence intervals for nonparametric functionals and bears a duality with the empirical likelihood (EL). In this paper, we show how adjusting the ball size of such type of DRO can reduce higher-order coverage errors similar to the so-called Bartlett correction. Our correction, which applies to general von Mises differentiable functionals, is more general than the existing EL literature that only focuses on smooth function models or M-estimation. Moreover, we demonstrate a higher-order “self-normalizing” property of DRO regardless of the choice of divergence. Our approach builds on the development of a higher-order expansion of DRO, which is obtained through an asymptotic analysis on a fixed-point equation arising from the Karush-Kuhn-Tucker conditions. Funding: This work was supported by the National Science Foundation, Division of Information and Intelligent Systems [Grant IIS-1849280] and the Division of Civil, Mechanical and Manufacturing Innovation [Grant CAREER CMMI-1834710].


  • Open Access

    Extreme conformal prediction: Reliable intervals for high-impact events

    by Olivier C. Pasche, Henry Lam, Sebastian Engelke

    Conformal prediction is a popular method to construct prediction intervals with marginal coverage guarantees from black-box machine learning models. In applications with potentially high-impact events, such as flooding or financial crises, regulators often require very high confidence for such intervals. However, if the desired level of confidence is too large relative to the amount of data used for calibration, then classical conformal methods provide infinitely wide, thus, uninformative prediction intervals. In this paper, we propose a new method to overcome this limitation. We bridge extreme value statistics and conformal prediction to provide reliable and informative prediction intervals with high-confidence coverage, which can be constructed using any black-box extreme quantile regression method. A weighted version of our approach can account for nonstationary data. The advantages of our extreme conformal prediction method are illustrated in a simulation study and in an application to flood risk forecasting.


  • On the Impossibility of Statistically Improving Empirical Optimization: A Second Order Stochastic Dominance Perspective

    by Henry Lam

    When the underlying probability distribution in a stochastic optimization is observed only through data, various data-driven formulations have been studied to obtain approximate optimal solutions. We show that no such formulations can, in a sense, theoretically improve the statistical quality of the solution obtained from empirical optimization. We argue this by proving that the first order behavior of the optimality gap against the oracle best solution, which includes both the bias and variance, for any data-driven solution second order stochastically dominates that from empirical optimization as long as suitable smoothness holds with respect to the underlying distribution. We demonstrate this impossibility of improvement in examples ranging across regularized optimization, distributionally robust optimization, parametric optimization, and Bayesian generalizations. We also discuss the connections of our results to other perspectives in statistics and data-driven optimization and illustrate practical implications in choosing among data-driven formulations. This paper was accepted by J. George Shanthikumar, data science. Funding: This work was supported by the National Science Foundation Division of Information and Intelligent Systems [Grant 1849280] and Division of Civil, Mechanical, and Manufacturing Innovation [Grant 1834710]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.04482 .


  • Open Access

    Calibrating Non-Identifiable High-Dimensional Simulation Models: A Framework via Eligibility Set

    by Yuanlu Bai, Tucker Balch, Haoxian Chen, Danial Dervovic, Henry Lam, Svitlana Vyetrenko

    Stochastic simulation aims to compute output performance for complex models that lack analytical tractability. To ensure accurate prediction, the model needs to be calibrated and validated against real data. Conventional methods approach these tasks by assessing the model-data match via simple hypothesis tests or distance minimization in an ad hoc fashion, but they can encounter challenges arising from non-identifiability and high dimensionality. In this article, we investigate a framework to develop calibration schemes that satisfies rigorous frequentist statistical guarantees, via a basic notion that we call eligibility set designed to bypass non-identifiability via a set-based estimation. We investigate a feature extraction-then-aggregation approach to construct these sets that target at multivariate outputs. We demonstrate our methodology on several numerical examples, including an application to calibration of a limit order book market simulator (ABIDES).


  • Convolution Bounds on Quantile Aggregation

    by Jose Blanchet, Henry Lam, Yang Liu, Ruodu Wang

    Advancing Risk Assessment: New Ways To Compute Quantile Aggregation This issue features a pivotal study on quantile aggregation amid dependence uncertainty, an area critical to finance, risk management, and statistics. The authors introduce “convolution bounds,” derived from a recent inf-convolution formula of quantiles and related risk measures. The obtained analytical tools unify existing results and enhance the understanding of quantile methods by providing general, sharp, and computationally efficient solutions. The results offer insights into the extremal dependence structures, with several implications in risk management and economic analysis applications. For more detailed insights, read the full paper, “Convolution Bounds on Quantile Aggregation” (reference: [insert reference]).


  • Distributionally Constrained Black-Box Stochastic Gradient Estimation and Optimization

    by Henry Lam, Junhui Zhang

    Constructing stochastic gradient estimators and descent algorithms is a core problem in stochastic optimization. In the black-box setting, namely when only noisy function evaluations are available, such gradient estimators are typically constructed via finite-differencing. In this regard, there have been schemes that aim to obtain gradient estimators for potentially many dimensions simultaneously via only few sample observations or simulation runs. However, for problems with probability simplex constraints, which arise in a range of applications from distributionally robust analysis to inverse model calibration, these schemes run into challenges one way or another when attempting to balance bias-variance in a constraint-compatible manner. This motivates us to create a new design framework for random perturbation generators and estimation schemes that bypass these challenges. Our culminated class of estimators, which is based on the Dirichlet mixtures, is demonstrably effective in distributionally constrained gradient estimation and optimization under various black-box settings.


  • Open Access

    DRO: A Python Library for Distributionally Robust Optimization in Machine Learning

    by Jiashuo Liu, Tianyu Wang, Henry Lam, Hongseok Namkoong, Jose Blanchet

    We introduce dro, an open-source Python library for distributionally robust optimization (DRO) for regression and classification problems. The library implements 14 DRO formulations and 9 backbone models, enabling 79 distinct DRO methods. Furthermore, dro is compatible with both scikit-learn and PyTorch. Through vectorization and optimization approximation techniques, dro reduces runtime by 10x to over 1000x compared to baseline implementations on large-scale datasets. Comprehensive documentation is available at https://python-dro.org.