Disparity between statistical and clinical significance in published randomised controlled trials indexed in PubMed: a protocol for a cross-sectional methodological survey
Open Access
Abstract
Introduction The commonly used frequentist paradigm of null hypothesis statistics testing with its reliance on the p-value and the corresponding notion of ‘statistical significance’ has been under ongoing criticism. Misinterpretation and misuse of the p-value have contributed to publication bias, unreliable studies, frequent false positives, fraud and mistrust in results of scientific studies. While p-values themselves are still useful, part of the problem may be the confusion between statistical and clinical significance. In randomised controlled trials of health interventions, this confusion could lead to erroneous conclusions about treatment efficacy, research waste and compromised patient outcomes. The extent to which clinical and statistical significance of published randomised clinical trials do not match is not known. This is a protocol for a methodological study to understand the extent of the problem of disparities between statistical and clinical significance in published clinical trials, and to identify and assess the factors associated with discrepant results in these studies. Methods and analysis A methodological survey of published randomised controlled trials is planned. Trials published between 2018 and 2022 and their protocols will be searched and screened for inclusion, with a planned sample size of 500 studies. The reported minimum clinically important difference, the study effect size and confidence intervals will be used to assess clinical importance of trial results. Comparison of statistical significance and clinical importance of the trial results will be used to determine disparity. Data will be analysed to estimate the outcomes, and factors associated with disparate study results will be assessed using logistic regression analysis. Ethics and dissemination Ethical approval for the study has been granted by Stellenbosch University’s Health Research Ethics Committee. This is part of a larger study towards a PhD in Biostatistics and will be disseminated as a thesis, conference abstract and peer-reviewed manuscript.Keywords
Funding Information
- African Centre for Biostatistical Excellence supported by the Fogarty International Centre of the National Institutes of Health (D43 TW010547)
This publication has 26 references indexed in Scilit:
- Assessing risk factors with information beyond P value thresholds: Statistical significance does not equal clinical importanceCancer, 2020
- Fragility of clinical trials across research fields: A synthesis of methodological reviewsContemporary Clinical Trials, 2020
- P-hacking in clinical trials and how incentives shape the distribution of results across phasesProceedings of the National Academy of Sciences, 2020
- The REDCap consortium: Building an international community of software platform partnersJournal of Biomedical Informatics, 2019
- Scientists rise up against statistical significanceNature, 2019
- Reproducibility of science: Fraud, impact factors and carelessnessJournal of Molecular and Cellular Cardiology, 2018
- The ASA Statement on p -Values: Context, Process, and PurposeThe American Statistician, 2016
- Statistical tests, P values, confidence intervals, and power: a guide to misinterpretationsEuropean Journal of Epidemiology, 2016
- Determination of the clinical importance of study resultsJournal of General Internal Medicine, 2002
- Evidence-based medicineSeminars in Perinatology, 1997