Researchers’ Perceptions of Statistical Significance Contribute to Bias in Health and Exercise Science
Authors:
- 14 April 2016
- journal article
- research article
- Published by Taylor & Francis in Measurement in Physical Education and Exercise Science
- Vol. 20 (3) , 131-139
Abstract
We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers’ decision making. Participants were presented with scenarios (testing a central hypothesis with p = .06 or p = .04) in a random order and surveyed about what they would do in each scenario. Participants showed significant bias in deciding to randomize additional participants when p = .06 (24.5%) compared to when p = .04 (6.5%), χ2 = 19.86, odds ratio (OR) = 13.5. Based on this bias to randomize additional participants, we present statistical simulations showing the cumulative negative effect this bias has on the false positive rate for a field. While it is important for researchers to have flexibility in their data-processing decisions, there is a cost/benefit trade-off to experimenter degrees of freedom and it is important that these decisions be unbiased. These data highlight the importance of a priori analysis plans in research.Keywords
This publication has 24 references indexed in Scilit:
- Sample Size and Power Estimates for a Confirmatory Factor Analytic Model in Exercise and SportResearch Quarterly for Exercise and Sport, 2011
- Confidence intervals permit, but don’t guarantee, better inference than statistical significance testingFrontiers in Psychology, 2010
- Comparison of Registered and Published Primary Outcomes in Randomized Controlled TrialsJAMA, 2009
- Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting BiasPLOS ONE, 2008
- Publication Bias in Empirical Sociological ResearchSociological Methods & Research, 2008
- Error bars in experimental biologyJournal of Cell Biology, 2007
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- Survival differences after stroke in a multiethnic population: follow-up study with the south London stroke registerBMJ, 2005
- The earth is round (p < .05).American Psychologist, 1994
- Consequences of prejudice against the null hypothesis.Psychological Bulletin, 1975