Reverse-Bayes analysis of two common misinterpretations of significance tests
Author:
- 17 January 2013
- journal article
- Published by SAGE Publications in Clinical Trials
- Vol. 10 (2) , 236-242
Abstract
Background Misunderstanding of significance tests and P values is widespread in clinical research and elsewhere. Purpose To assess the implications of two common mistakes in the interpretation of statistical significance tests. The first one is the misinterpretation of the type I error rate as the expected proportion of false-positive results among all those called significant, also known as the false-positive report probability (FPRP). The second is the misinterpretation of a P value as (posterior) probability of the null hypothesis. Methods A reverse-Bayes approach is used to calculate a lower bound on the proportion of truly effective treatments that would ensure the FPRP to be equal or below the type I error rate. A reverse-Bayes approach using minimum Bayes factors (BFs) yields upper bounds on the prior probability of the null hypothesis that would justify the interpretation of the P value as the posterior probability of the null hypothesis. Results In a typical clinical trials setting, more than 50% of the treatments need to be truly effective to justify equality of the type I error rate and the FPRP. To interpret the P value as posterior probability, the difference between the corresponding prior probability and the P value cannot exceed 12.4 percentage points. Limitations The first analysis requires that the (one-sided) type I error rate is smaller than the type II error rate. The second result is valid under different scenarios describing how to transform P values to minimum BFs. Conclusions The two misinterpretations imply strong and often unrealistic assumptions on the prior proportion or probability of truly effective treatments.Keywords
This publication has 31 references indexed in Scilit:
- Low P-values exclude nothing, and P-values are no substitute for measures of effectJournal of Clinical Epidemiology, 2011
- Demystify Statistical Significance--Time to Move on From the P Value to Bayesian AnalysisJNCI: Journal of the National Cancer Institute, 2010
- Re: The ongoing tyranny of statistical significance testing in biomedical researchEuropean Journal of Epidemiology, 2010
- The ongoing tyranny of statistical significance testing in biomedical researchEuropean Journal of Epidemiology, 2010
- A Dirty Dozen: Twelve P-Value MisconceptionsSeminars in Hematology, 2008
- Confusion Over Measures of Evidence (p's) Versus Errors (α's) in Classical Statistical TestingThe American Statistician, 2003
- Sifting the evidence---what's wrong with significance tests? Another comment on the role of statistical methodsBMJ, 2001
- That Confounded P-ValueEpidemiology, 1998
- The earth is round (p < .05).American Psychologist, 1994
- The role ofP‐values in analysing trial resultsStatistics in Medicine, 1993