Being Honest in Backtest Reporting: A Template for Disclosing Multiple Tests
- 31 October 2018
- journal article
- research article
- Published by With Intelligence LLC in The Journal of Portfolio Management
- Vol. 45 (1) , 141-147
Abstract
Selection bias under multiple testing is a serious problem. From a practitioner’s perspective, failure to disclose the impact of multiple tests of a proposed investment strategy to clients and senior management can lead to the adoption of a false discovery. Clients will lose money, senior management will misallocate resources, and the firm may be exposed to reputational, legal, and regulatory risks. From the perspective of academic journals that publish evidence supporting an investment strategy, the failure to address selection bias under multiple testing threatens to invalidate large portions of the literature in empirical finance. In this article, the authors propose a template that practitioners should use to fairly disclose multiple tests involved in an alleged discovery when pitching strategies to clients and senior management. The same template could be used by contributors to academic journals so that referees, and ultimately readers, can assess the strategy. By disclosing this information, those who are charged with making the final decision about a discovery can evaluate the probability that the purported discovery is false.Keywords
This publication has 6 references indexed in Scilit:
- … and the Cross-Section of Expected ReturnsThe Review of Financial Studies, 2015
- Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample PerformanceNotices of the American Mathematical Society, 2014
- Portfolio Performance Manipulation and Manipulation-proof Performance MeasuresThe Review of Financial Studies, 2007
- The Use of Statistics in Medical ResearchThe American Statistician, 2007
- Rectangular Confidence Regions for the Means of Multivariate Normal DistributionsJournal of the American Statistical Association, 1967
- IX. On the problem of the most efficient tests of statistical hypothesesPhilosophical Transactions of the Royal Society A, 1933