p-Hacking: Evidence from Two Million Trading Strategies
Authors:
Preprint
- 12 August 2017
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
We implement a data mining approach to generate about 2.1 million trading strategies. This large set of strategies serves as a laboratory to evaluate the seriousness of p-hacking and data snooping in finance. We apply multiple hypothesis testing techniques that account for cross-correlations in signals and returns to produce t-statistic thresholds that control the proportion of false discoveries. We find that the difference in rejections rates produced by single and multiple hypothesis testing is such that most rejections of the null of no outperformance under single hypothesis testing are likely false (i.e., we find a very high rate of type I errors). Combining statistical criteria with economic considerations, we find that a remarkably small number of strategies survive our thorough vetting procedure. Even these surviving strategies have no theoretical underpinnings. Overall, p-hacking is a serious problem and, correcting for it, outperforming trading strategies are rare.Keywords
All Related Versions
This publication has 43 references indexed in Scilit:
- Publication Bias and the Cross-Section of Stock ReturnsThe Review of Asset Pricing Studies, 2019
- Reassessing False Discoveries in Mutual Fund Performance: Skill, Luck, or Lack of Power?The Journal of Finance, 2019
- Do T-Stat Hurdles Need to Be Raised?: Direct Estimates of False Discoveries in the Cross-Section of Stock ReturnsSSRN Electronic Journal, 2018
- Comparing Asset Pricing ModelsThe Journal of Finance, 2018
- Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say 'Usually Not'SSRN Electronic Journal, 2015
- Have capital market anomalies attenuated in the recent era of high liquidity and trading activity?Journal of Accounting and Economics, 2014
- False Discoveries in Mutual Fund Performance: Measuring Luck in Estimated AlphasThe Journal of Finance, 2010
- Teoria Statistica Delle Classi e Calcolo Delle Probabilità2010
- On Persistence in Mutual Fund PerformanceThe Journal of Finance, 1997
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingJournal of the Royal Statistical Society Series B: Statistical Methodology, 1995