The Composite Overfit Analysis Framework: Assessing the Out-of-Sample Generalizability of Construct-Based Models Using Predictive Deviance, Deviance Trees, and Unstable Paths
- 1 January 2024
- journal article
- research article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 70 (1) , 647-669
Abstract
Construct-based models have become a mainstay of management and information systems research. However, these models are likely overfit to the data samples upon which they are estimated, making them risky to use in explanatory, prescriptive, or predictive ways outside a given sample. Empirical researchers currently lack tools to analyze why and how their models may not generalize out of sample. We propose a composite overfit analysis (COA) framework that applies predictive tools to describe the sources and ramifications of overfit in terms of the focal concepts important to empirical researchers: cases, constructs, and causal paths. The COA framework begins by using a leave-one-out crossvalidation procedure to identify cases with unusually high predictive error given their in-sample fit—a difference we describe as predictive deviance. The framework then employs a novel deviance tree method to group deviant cases that have similar predictive deviance and for similar theoretical reasons. We then employ a leave-deviant-group-out method, which sequentially analyzes how each deviant group affects model parameters, thereby identifying potentially unstable paths in the model. We can then infer descriptive reasons for why and how overfit affects a given model and data sample using the grouping criteria of the deviance tree, construct scores of deviant groups, and resulting unstable paths. These insights allow researchers to identify unexpected behavior that could define boundary conditions of their theory or point to new theoretical phenomena. We demonstrate the practical utility of our analytical framework on a technology adoption model in a new context. This paper was accepted by Dongjun Wu, information systems. Funding: This work was partially supported by the Ministry of Science and Technology, Taiwan [Grants 109-2811-H-007-503 and 108-2410-H-007-091-MY3] and by the 2021 Arts and Social Sciences Benefaction Fund of Trinity College Dublin, Ireland. Supplemental Material: The data files and online appendix are available at https://doi.org/10.1287/mnsc.2023.4705 .Keywords
This publication has 45 references indexed in Scilit:
- Rethinking Partial Least Squares Path Modeling: In Praise of Simple MethodsLong Range Planning, 2012
- Exceptional boards: Environmental experience and positive deviance from institutional normsJournal of Organizational Behavior, 2012
- Sensitivity Analysis in Structural Equation Models: Cases and Their InfluenceMultivariate Behavioral Research, 2011
- To Explain or to Predict?Statistical Science, 2010
- Variations of the kanban system: Literature review and classificationInternational Journal of Production Economics, 2010
- A survey of cross-validation procedures for model selectionStatistics Surveys, 2010
- Pursuing FailureOrganizational Research Methods, 2009
- Toward the Construct Definition of Positive DevianceAmerican Behavioral Scientist, 2004
- A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methodsBiometrika, 1989
- Distribution-free performance bounds for potential function rulesIEEE Transactions on Information Theory, 1979