Reproducibility in Management Science
Authors:
- Miloš FišarMFMiloš Fišar
- Ben GreinerBGBen Greiner
- Christoph HuberCHChristoph Huber
- Elena KatokEKElena Katok
- Ali I. OzkesAOAli I. Ozkes
- the Management Science Reproducibility CollaborationTCthe Management Science Reproducibility Collaboration
- 1 March 2024
- journal article
- research article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Management Science
- Vol. 70 (3) , 1343-1356
Abstract
With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019. When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%. These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility. Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness. This paper was accepted by David Simchi-Levi, behavioral economics and decision analysis–fast track. Supplemental Material: The online appendices and data are available at https://doi.org/10.1287/mnsc.2023.03556 .Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- Estimating the reproducibility of psychological scienceScience, 2015
- WHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?Journal of Economic Surveys, 2013
- Scientific UtopiaPerspectives on Psychological Science, 2012
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth TellingPsychological Science, 2012
- False-Positive PsychologyPsychological Science, 2011
- Lessons from the JMCB ArchiveJournal of Money, Credit and Banking, 2006
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- Academic economists behaving badly? A survey on three areas of unethical behaviorEconomic Inquiry, 2001
- Are all Economic Hypotheses False?Journal of Political Economy, 1992
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955