Diagnosing Model Performance Under Distribution Shift
Top Cited Papers
- 1 March 2026
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Operations Research
- Vol. 74 (2) , 898-916
Abstract
Diagnosing Why Models Fail Under Distribution Shift and What to Do Next Predictive models often perform worse when deployed in a new target setting, but it is rarely clear why. In “Diagnosing Model Performance Under Distribution Shift,” Cai, Namkoong, and Yadlowsky introduce a diagnostic, distribution shift decomposition (DISDE), that attributes the change in performance from the training to target distributions into terms for (i) an increase in harder but previously seen inputs from training, (ii) changes in how outcomes relate to inputs, and (iii) poor performance on new input regions absent from the training data. Applications to employment prediction demonstrate how this decomposition can inform potential modeling improvements, guiding whether to use domain adaptation techniques, adjust model covariates, or collect new samples. Additionally, DISDE is used to help explain why certain domain adaptation methods fail to improve model performance for satellite image classification.Keywords
All Related Versions
This publication has 70 references indexed in Scilit:
- External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized PatientsJAMA Internal Medicine, 2021
- Balancing vs modeling approaches to weighting in practiceStatistics in Medicine, 2020
- Data-driven robust optimizationMathematical Programming, 2017
- Twenty years post‐NIH Revitalization Act: Enhancing minority participation in clinical trials (EMPaCT): Laying the groundwork for improving minority clinical trial accrualCancer, 2014
- Design of Observational Studies2010
- Adapting Visual Category Models to New Domains2010
- A warm-start approach for large-scale stochastic linear programsMathematical Programming, 2009
- Introduction to Nonparametric Estimation2009
- Stochastic Simulation: Algorithms and Analysis2007
- Semiparametric efficiency boundsJournal of Applied Econometrics, 1990