Data splitting as a countermeasure against hypothesis fishing: with a case study of predictors for low back pain
Authors:
Open Access
- 21 February 2008
- journal article
- research article
- Published by Springer Nature in European Journal of Epidemiology
- Vol. 23 (4) , 237-242
Abstract
There is growing concern in the scientific community that many published scientific findings may represent spurious patterns that are not reproducible in independent data sets. A reason for this is that significance levels or confidence intervals are often applied to secondary variables or sub-samples within the trial, in addition to the primary hypotheses (multiple hypotheses). This problem is likely to be extensive for population-based surveys, in which epidemiological hypotheses are derived after seeing the data set (hypothesis fishing). We recommend a data-splitting procedure to counteract this methodological problem, in which one part of the data set is used for identifying hypotheses, and the other is used for hypothesis testing. The procedure is similar to two-stage analysis of microarray data. We illustrate the process using a real data set related to predictors of low back pain at 14-year follow-up in a population initially free of low back pain. “Widespreadness” of pain (pain reported in several other places than the low back) was a statistically significant predictor, while smoking was not, despite its strong association with low back pain in the first half of the data set. We argue that the application of data splitting, in which an independent party handles the data set, will achieve for epidemiological surveys what pre-registration has done for clinical studies.Keywords
This publication has 6 references indexed in Scilit:
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- Two‐Stage Designs for Gene–Disease Association StudiesBiometrics, 2002
- The Elements of Statistical Learning2001
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingJournal of the Royal Statistical Society Series B: Statistical Methodology, 1995
- Model Selection and Accounting for Model Uncertainty in Graphical Models Using Occam's WindowJournal of the American Statistical Association, 1994
- Standardised Nordic questionnaires for the analysis of musculoskeletal symptomsApplied Ergonomics, 1987