How hard is it to pick the right model? MCS and backtest overfitting
- 21 June 2018
- journal article
- research article
- Published by SAGE Publications in Algorithmic Finance
- Vol. 7 (1-2) , 53-61
Abstract
Recent advances in machine learning, artificial intelligence, and the availability of billions of high frequency data signals have made model selection a challenging and pressing need. However, most of the model selection methods available in modern finance are subject to backtest overfitting. This is the probability that one will select a financial strategy that outperforms during backtest, but underperforms in practice. We evaluate the performance of the novel model confidence set (MCS) introduced in Hansen et al. (2011a) in a simple machine learning trading strategy problem. We find that MCS is not robust to multiple testing and that it requires a very high signal-to-noise ratio to be utilizable. More generally, we raise awareness on the limitations of model selection in finance.Keywords
This publication has 31 references indexed in Scilit:
- Measuring the Prevalence of Questionable Research Practices With Incentives for Truth TellingPsychological Science, 2012
- Do the Best Hedge Funds Hedge?The Review of Financial Studies, 2010
- Hypothesis Testing in EconometricsAnnual Review of Economics, 2010
- The Model Confidence SetSSRN Electronic Journal, 2010
- False Discoveries in Mutual Fund Performance: Measuring Luck in Estimated AlphasThe Journal of Finance, 2010
- Computational Social ScienceScience, 2009
- FORMALIZED DATA SNOOPING BASED ON GENERALIZED ERROR RATESEconometric Theory, 2007
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- A Direct Approach to False Discovery RatesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2002
- The Statistics of Sharpe RatiosCFA Magazine, 2002