Hongseok Namkoong
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Hongseok Namkoong has a total of 62 co-authors. Here are the top co-authors by number of shared publications:
Ethan Che
Columbia Business School
3 shared publications- Jiashuo Liu
Tsinghua University, Beijing, China
3 shared publications - John Duchi
Department of Statistics Stanford University
3 shared publications - Steve Yadlowsky
Department of Statistics 1 Decision, Risk Operations Division 2 Team Brain
3 shared publications - Ariel Boyarsky
Decision, Risk, and Operations Division Columbia Business School
2 shared publications Daksh Mittal
Decision, Risk, and Operations Division, Columbia Business School, New York, New York 10027
2 shared publications
Latest Publications
- 13 November 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
Rethinking Distribution Shifts: Empirical Analysis and Modeling for Tabular Data
Different distribution shifts require different interventions, and algorithms must be grounded in the specific shifts they address. However, methodological development for robust algorithms typically relies on structural assumptions that lack empirical validation. Advocating for an empirically grounded data-driven approach to algorithm development, we build an empirical testbed comprising natural shifts across 8 tabular data sets, 172 distribution pairs over 45 methods, and 90,000 method configurations encompassing empirical risk minimization and distributionally robust optimization (DRO) methods. We find that Y|X-shifts are most prevalent in our testbed, in stark contrast to the heavy focus on X (covariate)-shifts in the machine learning literature, and that the performance of robust algorithms is no better than that of vanilla methods. To understand why, we conduct an in-depth empirical analysis of DRO methods and find that underlooked implementation details—such as the choice of underlying model class (e.g., LightGBM) and hyperparameter selection—have a bigger impact on performance than the ambiguity set or its radius. We illustrate via case studies how a data-driven understanding of distribution shifts can provide a new approach to algorithm development. This paper was accepted by Wolfram Wiesemann, data science. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.08298 .
- 13 November 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
Rethinking Distribution Shifts: Empirical Analysis and Modeling for Tabular Data
Different distribution shifts require different interventions, and algorithms must be grounded in the specific shifts they address. However, methodological development for robust algorithms typically relies on structural assumptions that lack empirical validation. Advocating for an empirically grounded data-driven approach to algorithm development, we build an empirical testbed comprising natural shifts across 8 tabular data sets, 172 distribution pairs over 45 methods, and 90,000 method configurations encompassing empirical risk minimization and distributionally robust optimization (DRO) methods. We find that Y|X-shifts are most prevalent in our testbed, in stark contrast to the heavy focus on X (covariate)-shifts in the machine learning literature, and that the performance of robust algorithms is no better than that of vanilla methods. To understand why, we conduct an in-depth empirical analysis of DRO methods and find that underlooked implementation details—such as the choice of underlying model class (e.g., LightGBM) and hyperparameter selection—have a bigger impact on performance than the ambiguity set or its radius. We illustrate via case studies how a data-driven understanding of distribution shifts can provide a new approach to algorithm development. This paper was accepted by Wolfram Wiesemann, data science. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.08298 .
- 19 August 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Mathematics of Operations Research
Evaluating Model Performance Under Worst-Case Subpopulations
The performance of machine learning models degrades when the training population is different from that seen under operation. Toward assessing distributional robustness, we study the worst-case performance of a model over all subpopulations of a given size defined with respect to core attributes Z. This notion of robustness can consider arbitrary (continuous) attributes Z and automatically accounts for complex intersectionality in disadvantaged groups. We develop a scalable yet principled two-stage estimation procedure that can evaluate the robustness of state-of-the-art models. We prove that our procedure enjoys several finite-sample convergence guarantees, including dimension-free convergence. Instead of overly conservative notions based on Rademacher complexities, our evaluation error depends on the dimension of Z only through the out-of-sample error in estimating the performance conditional on Z. On real data sets, we demonstrate that our method certifies the robustness of a model and prevents deployment of unreliable models.
- 19 August 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Mathematics of Operations Research
Evaluating Model Performance Under Worst-Case Subpopulations
The performance of machine learning models degrades when the training population is different from that seen under operation. Toward assessing distributional robustness, we study the worst-case performance of a model over all subpopulations of a given size defined with respect to core attributes Z. This notion of robustness can consider arbitrary (continuous) attributes Z and automatically accounts for complex intersectionality in disadvantaged groups. We develop a scalable yet principled two-stage estimation procedure that can evaluate the robustness of state-of-the-art models. We prove that our procedure enjoys several finite-sample convergence guarantees, including dimension-free convergence. Instead of overly conservative notions based on Rademacher complexities, our evaluation error depends on the dimension of Z only through the out-of-sample error in estimating the performance conditional on Z. On real data sets, we demonstrate that our method certifies the robustness of a model and prevents deployment of unreliable models.
- Top Cited Papers
- 1 March 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Operations Research
- Vol. 74 (2) , 898-916
Diagnosing Model Performance Under Distribution Shift
Diagnosing Why Models Fail Under Distribution Shift and What to Do Next Predictive models often perform worse when deployed in a new target setting, but it is rarely clear why. In “Diagnosing Model Performance Under Distribution Shift,” Cai, Namkoong, and Yadlowsky introduce a diagnostic, distribution shift decomposition (DISDE), that attributes the change in performance from the training to target distributions into terms for (i) an increase in harder but previously seen inputs from training, (ii) changes in how outcomes relate to inputs, and (iii) poor performance on new input regions absent from the training data. Applications to employment prediction demonstrate how this decomposition can inform potential modeling improvements, guiding whether to use domain adaptation techniques, adjust model covariates, or collect new samples. Additionally, DISDE is used to help explain why certain domain adaptation methods fail to improve model performance for satellite image classification.
- 1 March 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Operations Research
- Vol. 74 (2) , 898-916
Top Cited PapersDiagnosing Model Performance Under Distribution Shift
Diagnosing Why Models Fail Under Distribution Shift and What to Do Next Predictive models often perform worse when deployed in a new target setting, but it is rarely clear why. In “Diagnosing Model Performance Under Distribution Shift,” Cai, Namkoong, and Yadlowsky introduce a diagnostic, distribution shift decomposition (DISDE), that attributes the change in performance from the training to target distributions into terms for (i) an increase in harder but previously seen inputs from training, (ii) changes in how outcomes relate to inputs, and (iii) poor performance on new input regions absent from the training data. Applications to employment prediction demonstrate how this decomposition can inform potential modeling improvements, guiding whether to use domain adaptation techniques, adjust model covariates, or collect new samples. Additionally, DISDE is used to help explain why certain domain adaptation methods fail to improve model performance for satellite image classification.
- 1 January 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Operations Research
- Vol. 74 (1) , 464-483
Minimax Optimal Estimation of Stability Under Distribution Shift
How to Efficiently Estimate Stability Measures: A Minimax Optimal Approach In “Minimax Optimal Estimation of Stability Under Distribution Shift,” Hongseok Namkoong, Yuanzhe Ma, and Peter W. Glynn address the challenge of benchmarking the performance of decision policies and prediction models under distribution shift. Conventional risk measures and distributionally robust losses typically require specifying the magnitude of possible distribution shift—a quantity that is difficult to determine in practice. Instead, the authors consider a stability measure defined in terms of the acceptable level of performance degradation, which is more intuitive. To efficiently estimate this measure, they consider an estimator based on the dual formulation of the stability measure and show that this estimator is minimax optimal. Their results quantify the convergence rate of the estimator, which exhibits a fundamental phase shift behavior. In addition, they empirically observe that the stability measure reliably captures system performance under distribution shift in applications including queueing systems and healthcare prediction tasks.
- 1 January 2026
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Operations Research
- Vol. 74 (1) , 464-483
Minimax Optimal Estimation of Stability Under Distribution Shift
How to Efficiently Estimate Stability Measures: A Minimax Optimal Approach In “Minimax Optimal Estimation of Stability Under Distribution Shift,” Hongseok Namkoong, Yuanzhe Ma, and Peter W. Glynn address the challenge of benchmarking the performance of decision policies and prediction models under distribution shift. Conventional risk measures and distributionally robust losses typically require specifying the magnitude of possible distribution shift—a quantity that is difficult to determine in practice. Instead, the authors consider a stability measure defined in terms of the acceptable level of performance degradation, which is more intuitive. To efficiently estimate this measure, they consider an estimator based on the dual formulation of the stability measure and show that this estimator is minimax optimal. Their results quantify the convergence rate of the estimator, which exhibits a fundamental phase shift behavior. In addition, they empirically observe that the stability measure reliably captures system performance under distribution shift in applications including queueing systems and healthcare prediction tasks.
- Open Access
A Broader View of Thompson Sampling
Thompson Sampling is one of the most widely used and studied bandit algorithms, known for its simple structure, low regret performance, and solid theoretical guarantees. Yet, in stark contrast to most other families of bandit algorithms, the exact mechanism through which posterior sampling (as introduced by Thompson) is able to "properly" balance exploration and exploitation, remains a mystery. In this paper we show that the core insight to address this question stems from recasting Thompson Sampling as an online optimization algorithm. To distill this, a key conceptual tool is introduced, which we refer to as "faithful" stationarization of the regret formulation. Essentially, the finite horizon dynamic optimization problem is converted into a stationary counterpart which "closely resembles" the original objective (in contrast, the classical infinite horizon discounted formulation, that leads to the Gittins index, alters the problem and objective in too significant a manner). The newly crafted time invariant objective can be studied using Bellman's principle which leads to a time invariant optimal policy. When viewed through this lens, Thompson Sampling admits a simple online optimization form that mimics the structure of the Bellman-optimal policy, and where greediness is regularized by a measure of residual uncertainty based on point-biserial correlation. This answers the question of how Thompson Sampling balances exploration-exploitation, and moreover, provides a principled framework to study and further improve Thompson's original idea.
Open AccessA Broader View of Thompson Sampling
Thompson Sampling is one of the most widely used and studied bandit algorithms, known for its simple structure, low regret performance, and solid theoretical guarantees. Yet, in stark contrast to most other families of bandit algorithms, the exact mechanism through which posterior sampling (as introduced by Thompson) is able to "properly" balance exploration and exploitation, remains a mystery. In this paper we show that the core insight to address this question stems from recasting Thompson Sampling as an online optimization algorithm. To distill this, a key conceptual tool is introduced, which we refer to as "faithful" stationarization of the regret formulation. Essentially, the finite horizon dynamic optimization problem is converted into a stationary counterpart which "closely resembles" the original objective (in contrast, the classical infinite horizon discounted formulation, that leads to the Gittins index, alters the problem and objective in too significant a manner). The newly crafted time invariant objective can be studied using Bellman's principle which leads to a time invariant optimal policy. When viewed through this lens, Thompson Sampling admits a simple online optimization form that mimics the structure of the Bellman-optimal policy, and where greediness is regularized by a measure of residual uncertainty based on point-biserial correlation. This answers the question of how Thompson Sampling balances exploration-exploitation, and moreover, provides a principled framework to study and further improve Thompson's original idea.
- Open Access
FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks -- Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation -- closely reproduce real-world financial analyst workflows. To ensure difficulty and reliability, we engage 70 professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes 635 questions spanning global and Greater China markets, and we evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly.By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning.
Open AccessFinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
Search has emerged as core infrastructure for LLM-based agents and is widely viewed as critical on the path toward more general intelligence. Finance is a particularly demanding proving ground: analysts routinely conduct complex, multi-step searches over time-sensitive, domain-specific data, making it ideal for assessing both search proficiency and knowledge-grounded reasoning. Yet no existing open financial datasets evaluate data searching capability of end-to-end agents, largely because constructing realistic, complicated tasks requires deep financial expertise and time-sensitive data is hard to evaluate. We present FinSearchComp, the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. FinSearchComp comprises three tasks -- Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation -- closely reproduce real-world financial analyst workflows. To ensure difficulty and reliability, we engage 70 professional financial experts for annotation and implement a rigorous multi-stage quality-assurance pipeline. The benchmark includes 635 questions spanning global and Greater China markets, and we evaluate 21 models (products) on it. Grok 4 (web) tops the global subset, approaching expert-level accuracy. DouBao (web) leads on the Greater China subset. Experimental analyses show that equipping agents with web search and financial plugins substantially improves results on FinSearchComp, and the country origin of models and tools impact performance significantly.By aligning with realistic analyst tasks and providing end-to-end evaluation, FinSearchComp offers a professional, high-difficulty testbed for complex financial search and reasoning.
- Open Access
Data-Driven Stochastic Modeling Using Autoregressive Sequence Models: Translating Event Tables to Queueing Dynamics
While queueing network models are powerful tools for analyzing service systems, they traditionally require substantial human effort and domain expertise to construct. To make this modeling approach more scalable and accessible, we propose a data-driven framework for queueing network modeling and simulation based on autoregressive sequence models trained on event-stream data. Instead of explicitly specifying arrival processes, service mechanisms, or routing logic, our approach learns the conditional distributions of event types and event times, recasting the modeling task as a problem of sequence distribution learning. We show that Transformer-style architectures can effectively parameterize these distributions, enabling automated construction of high-fidelity simulators. As a proof of concept, we validate our framework on event tables generated from diverse queueing networks, showcasing its utility in simulation, uncertainty quantification, and counterfactual evaluation. Leveraging advances in artificial intelligence and the growing availability of data, our framework takes a step toward more automated, data-driven modeling pipelines to support broader adoption of queueing network models across service domains.
Open AccessData-Driven Stochastic Modeling Using Autoregressive Sequence Models: Translating Event Tables to Queueing Dynamics
While queueing network models are powerful tools for analyzing service systems, they traditionally require substantial human effort and domain expertise to construct. To make this modeling approach more scalable and accessible, we propose a data-driven framework for queueing network modeling and simulation based on autoregressive sequence models trained on event-stream data. Instead of explicitly specifying arrival processes, service mechanisms, or routing logic, our approach learns the conditional distributions of event types and event times, recasting the modeling task as a problem of sequence distribution learning. We show that Transformer-style architectures can effectively parameterize these distributions, enabling automated construction of high-fidelity simulators. As a proof of concept, we validate our framework on event tables generated from diverse queueing networks, showcasing its utility in simulation, uncertainty quantification, and counterfactual evaluation. Leveraging advances in artificial intelligence and the growing availability of data, our framework takes a step toward more automated, data-driven modeling pipelines to support broader adoption of queueing network models across service domains.
- Open Access
DRO: A Python Library for Distributionally Robust Optimization in Machine Learning
We introduce dro, an open-source Python library for distributionally robust optimization (DRO) for regression and classification problems. The library implements 14 DRO formulations and 9 backbone models, enabling 79 distinct DRO methods. Furthermore, dro is compatible with both scikit-learn and PyTorch. Through vectorization and optimization approximation techniques, dro reduces runtime by 10x to over 1000x compared to baseline implementations on large-scale datasets. Comprehensive documentation is available at https://python-dro.org.
Open AccessDRO: A Python Library for Distributionally Robust Optimization in Machine Learning
We introduce dro, an open-source Python library for distributionally robust optimization (DRO) for regression and classification problems. The library implements 14 DRO formulations and 9 backbone models, enabling 79 distinct DRO methods. Furthermore, dro is compatible with both scikit-learn and PyTorch. Through vectorization and optimization approximation techniques, dro reduces runtime by 10x to over 1000x compared to baseline implementations on large-scale datasets. Comprehensive documentation is available at https://python-dro.org.
- Open Access
LLM Embeddings Improve Test-time Adaptation to Tabular $Y|X$-Shifts
For tabular datasets, the change in the relationship between the label and covariates ($Y|X$-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples. We focus on building more informative representations of tabular data that can mitigate $Y|X$-shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies. The code is available at https://github.com/namkoong-lab/LLM-Tabular-Shifts.
Open AccessLLM Embeddings Improve Test-time Adaptation to Tabular $Y|X$-Shifts
For tabular datasets, the change in the relationship between the label and covariates ($Y|X$-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples. We focus on building more informative representations of tabular data that can mitigate $Y|X$-shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies. The code is available at https://github.com/namkoong-lab/LLM-Tabular-Shifts.
- Open Access
QGym: Scalable Simulation and Benchmarking of Queuing Network Controllers
Queuing network control determines the allocation of scarce resources to manage congestion, a fundamental problem in manufacturing, communications, and healthcare. Compared to standard RL problems, queueing problems are distinguished by unique challenges: i) a system operating in continuous time, ii) high stochasticity, and iii) long horizons over which the system can become unstable (exploding delays). To spur methodological progress tackling these challenges, we present an open-sourced queueing simulation framework, QGym, that benchmark queueing policies across realistic problem instances. Our modular framework allows the researchers to build on our initial instances, which provide a wide range of environments including parallel servers, criss-cross, tandem, and re-entrant networks, as well as a realistically calibrated hospital queuing system. QGym makes it easy to compare multiple policies, including both model-free RL methods and classical queuing policies. Our testbed complements the traditional focus on evaluating algorithms based on mathematical guarantees in idealized settings, and significantly expands the scope of empirical benchmarking in prior work. QGym code is open-sourced at https://github.com/namkoong-lab/QGym.
Open AccessQGym: Scalable Simulation and Benchmarking of Queuing Network Controllers
Queuing network control determines the allocation of scarce resources to manage congestion, a fundamental problem in manufacturing, communications, and healthcare. Compared to standard RL problems, queueing problems are distinguished by unique challenges: i) a system operating in continuous time, ii) high stochasticity, and iii) long horizons over which the system can become unstable (exploding delays). To spur methodological progress tackling these challenges, we present an open-sourced queueing simulation framework, QGym, that benchmark queueing policies across realistic problem instances. Our modular framework allows the researchers to build on our initial instances, which provide a wide range of environments including parallel servers, criss-cross, tandem, and re-entrant networks, as well as a realistically calibrated hospital queuing system. QGym makes it easy to compare multiple policies, including both model-free RL methods and classical queuing policies. Our testbed complements the traditional focus on evaluating algorithms based on mathematical guarantees in idealized settings, and significantly expands the scope of empirical benchmarking in prior work. QGym code is open-sourced at https://github.com/namkoong-lab/QGym.