Stupid Data Miner Tricks
- 28 February 2007
- journal article
- Published by With Intelligence LLC in The Journal of Investing
- Vol. 16 (1) , 15-22
Abstract
This article originated over ten years ago as a set of joke slides showing silly spurious correlations. These statistically appealing relationships between the stock market and diary products and third world livestock populations have been cited often, in Business Week, the Wall Street Journal, the book “A Mathematician Looks at the Stock Market,” and elsewhere. Students from Bill Sharpe's classes at Stanford seem to be familiar with them. The slides were expanded to include some actual content about data mining, and reissued as an academic working paper in 2001. Occasional requests arrive from distant corners of the world, so I thank the editors of the Journal of Investing for publishing this article. Without taking a hatchet to the original, the advice offered remains valuable, perhaps even more so now that there is so much more data to mine. Monthly data arrives as a single data point, once a month. It's hard to avoid data mining sins if you look twice. Ticks, quotes, and executions arrive in millions ...Keywords
This publication has 4 references indexed in Scilit:
- Data Mining: Statistics and More?The American Statistician, 1998
- Behind the Smoke and Mirrors: Gauging the Integrity of Investment SimulationsCFA Magazine, 1992
- Data-Snooping Biases in Tests of Financial Asset Pricing ModelsThe Review of Financial Studies, 1990
- Selection Models and the File Drawer ProblemStatistical Science, 1988