Trends in the distribution of P values in epidemiology journals: a statistical, P -curve, and simulation study
- 21 August 2025
- journal article
- Published by Oxford University Press (OUP) in American Journal of Epidemiology
- Vol. 194 (12) , 3630-3639
Abstract
Epidemiologists have advocated for reporting confidence intervals and deemphasizing P values to address long-standing concerns about null-hypothesis statistical-significance testing, P hacking, and reproducibility. It is unknown if efforts to reduce reliance on P values have altered the distribution of P values. For 21 332 abstracts published 2000 to 2024 in 4 major epidemiology journals, two-sided P values (N = 25 288) were calculated from estimates and confidence intervals scraped using ChatGPT’s 4o-mini model. We evaluated trends over time to determine whether the empirical distribution of P values changed. We fitted to expected P-value distributions and simulated these distributions with and without assuming changes in statistical power over time. Average P values decreased from 2000 to 2024; the fraction of P values just below the .05 threshold also decreased. Fits to models indicate that statistical power increased. Increasing power would reduce average P value while also decreasing P values near the .05 threshold—precisely the trends observed in epidemiology journals. Although the frequency of P values near the .05 threshold has declined modestly, this likely reflects increases in statistical power rather than decreases in P hacking.Keywords
Funding Information
- NIH NIA (P01AG082653, F31AG085965, K99AG083121, F99AG083306, K00AG068431, K99AG088369, R00AG073454)
This publication has 33 references indexed in Scilit:
- Some properties of p-curves, with an application to gradual publication bias.Psychological Methods, 2018
- The Replication Crisis in Epidemiology: Snowball, Snow Job, or Winter Solstice?Current Epidemiology Reports, 2018
- Statistical inference in abstracts of major medical and epidemiology journals 1975–2014: a systematic reviewEuropean Journal of Epidemiology, 2016
- Disengaging from statistical significanceEuropean Journal of Epidemiology, 2016
- Assessment of vibration of effects due to model specification can demonstrate the instability of observational associationsJournal of Clinical Epidemiology, 2015
- P-curve: A key to the file-drawer.Journal of Experimental Psychology: General, 2014
- Statistical and clinical significance, and how to use confidence intervals to help interpret bothAustralian Critical Care, 2010
- The ongoing tyranny of statistical significance testing in biomedical researchEuropean Journal of Epidemiology, 2010
- Statistical methods in psychology journals: Guidelines and explanations.American Psychologist, 1999
- Mathematical vs. scientific significance.Psychological Bulletin, 1919