Case Study: Evaluation of a meta-analysis of the association between soy protein and cardiovascular disease
Preprint
Abstract
It is well-known that claims coming from observational studies most often fail to replicate. Experimental (randomized) trials, where conditions are under researcher control, have a high reputation and meta-analysis of experimental trials are considered the best possible evidence. Given the irreproducibility crisis, experiments lately are starting to be questioned. There is a need to know the reliability of claims coming from randomized trials. A case study is presented here independently examining a published meta-analysis of randomized trials claiming that soy protein intake improves cardiovascular health. Counting and p-value plotting techniques (standard p-value plot, p-value expectation plot, and volcano plot) are used. Counting (search space) analysis indicates that reported p-values from the meta-analysis could be biased low due to multiple testing and multiple modeling. Plotting techniques used to visualize the behavior of the data set used for meta-analysis suggest that statistics drawn from the base papers do not satisfy key assumptions of a random-effects meta-analysis. These assumptions include using unbiased statistics all drawn from the same population. Also, publication bias is unaddressed in the meta-analysis. The claim that soy protein intake should improve cardiovascular health is not supported by our analysis.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Data integrity of 35 randomised controlled trials in women’ healthEuropean Journal of Obstetrics & Gynecology and Reproductive Biology, 2020
- RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysisThe Lancet, 2020
- Statistical Inference and the Replication CrisisReview of Philosophy and Psychology, 2018
- Statistically Significant Differences in Composition of Petroleum Crude Oils Revealed by Volcano Plots Generated from Ultrahigh Resolution Fourier Transform Ion Cyclotron Resonance Mass SpectraEnergy & Fuels, 2018
- How does one detect scientific fraud – but avoid false accusations?Journal of Clinical Epidemiology, 2017
- Air quality environmental epidemiology studies are unreliableRegulatory Toxicology and Pharmacology, 2017
- Evolution of ReportingPValues in the Biomedical Literature, 1990-2015JAMA, 2016
- P-curve: A key to the file-drawer.Journal of Experimental Psychology: General, 2014
- Meta-analysis in clinical trialsControlled Clinical Trials, 1986
- Consequences of prejudice against the null hypothesis.Psychological Bulletin, 1975