Statistics in Service of Metascience: Measuring Replication Distance with Reproducibility Rate
Open Access
Abstract
Motivated by the recent putative reproducibility crisis, we discuss the relationship between the replicability of scientific studies, the reproducibility of results obtained in these replications, and the philosophy of statistics. Our approach focuses on challenges in specifying scientific studies for scientific inference via statistical inference and is complementary to classical discussions in the philosophy of statistics. We particularly consider the challenges in replicating studies exactly, using the notion of the idealized experiment. We argue against treating reproducibility as an inherently desirable property of scientific results, and in favor of viewing it as a tool to measure the distance between an original study and its replications. To sensibly study the implications of replicability and results reproducibility on inference, such a measure of replication distance is needed. We present an effort to delineate such a framework here, addressing some challenges in capturing the components of scientific studies while identifying others as ongoing issues. We illustrate our measure of replication distance by simulations using a toy example. Rather than replications, we present purposefully planned modifications as an appropriate tool to inform scientific inquiry. Our ability to measure replication distance serves scientists in their search for replication-ready studies. We believe that likelihood-based and evidential approaches may play a critical role towards building statistics that effectively serve the practical needs of science.Keywords
All Related Versions
Funding Information
- University of Idaho Open Access Publishing Fund (P20GM104420)
- National Institute of General Medical Sciences of the National Institutes of Health (P20GM104420)
This publication has 44 references indexed in Scilit:
- Rigorous exploration in a model-centric science via epistemic iteration.Journal of Applied Research in Memory and Cognition, 2023
- Modeling and Learning From Variation and CovariationJournal of the American Statistical Association, 2022
- Reconsidering statistical methods for assessing replication.Psychological Methods, 2021
- Comparing meta-analyses and preregistered multiple-laboratory replication projectsNature Human Behaviour, 2019
- Metastudies for robust tests of theoryProceedings of the National Academy of Sciences, 2018
- The natural selection of bad scienceRoyal Society Open Science, 2016
- Evidential statistics as a statistical modern synthesis to support 21st century sciencePopulation Ecology, 2015
- An iterative Monte Carlo method for nonconjugate Bayesian analysisStatistics and Computing, 1991
- Science and StatisticsJournal of the American Statistical Association, 1976
- On the Foundations of Statistical InferenceJournal of the American Statistical Association, 1962