Statistics in service of metascience: Measuring replication distance with reproducibility rate
Preprint
Abstract
Motivated by the recent putative reproducibility crisis, we discuss the relationship between replicability of scientific studies, reproducibility of results obtained in these replications, and the philosophy of statistics. Our approach focuses on challenges in specifying scientific studies for scientific inference via statistical inference, and is complementary to classical discussions in philosophy of statistics. We particularly consider the challenges in replicating studies exactly, using the notion of the idealized experiment. We argue against treating reproducibility as an inherently desirable property of scientific results, and in favor of viewing it as a tool to measure distance between an original study and its replications. To sensibly study the implications of replicability and results reproducibility on inference, such a measure of replication distance is needed. We present an effort to delineate such a framework here, addressing some challenges in capturing the components of scientific studies while identifying others as ongoing issues. We illustrate our measure of replication distance by simulations using a toy example. Rather than replications, we present purposefully planned modifications as an appropriate tool to inform scientific inquiry. Our ability to measure replication distance serves scientists in their search for replication-ready studies. We believe that likelihood-based and evidential approaches may play a critical role towards building a statistics that effectively serves the practical needs of science.Keywords
All Related Versions
- Published version: Entropy, 26 (10), 842.
This publication has 39 references indexed in Scilit:
- Rigorous exploration in a model-centric science via epistemic iteration.Journal of Applied Research in Memory and Cognition, 2023
- The logical structure of experiments lays the foundation for a theory of reproducibilityRoyal Society Open Science, 2023
- Modeling and Learning From Variation and CovariationJournal of the American Statistical Association, 2022
- The case for formal methodology in scientific reformRoyal Society Open Science, 2021
- Reconsidering statistical methods for assessing replication.Psychological Methods, 2021
- Metastudies for robust tests of theoryProceedings of the National Academy of Sciences, 2018
- The natural selection of bad scienceRoyal Society Open Science, 2016
- Evidential statistics as a statistical modern synthesis to support 21st century sciencePopulation Ecology, 2015
- An iterative Monte Carlo method for nonconjugate Bayesian analysisStatistics and Computing, 1991
- DO WE SEE THROUGH A MICROSCOPE?Pacific Philosophical Quarterly, 1981