Preserving Statistical Validity in Adaptive Data Analysis
Authors:
Open Access
- 14 June 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 117-126
Abstract
A great deal of effort has been devoted to reducing the risk of spurious scientific discoveries, from the use of sophisticated validation techniques, to deep statistical methods for controlling the false discovery rate in multiple hypothesis testing. However, there is a fundamental disconnect between the theoretical results and the practice of data analysis: the theory of statistical inference assumes a fixed collection of hypotheses to be tested, or learning algorithms to be applied, selected non-adaptively before the data are gathered, whereas in practice data is shared and reused with hypotheses and new analyses being generated on the basis of data exploration and the outcomes of previous analyses. In this work we initiate a principled study of how to guarantee the validity of statistical inference in adaptive data analysis. As an instance of this problem, we propose and investigate the question of estimating the expectations of m adaptively chosen functions on an unknown distribution given n random samples. We show that, surprisingly, there is a way to estimate an exponential in n number of expectations accurately even if the functions are chosen adaptively. This gives an exponential improvement over standard empirical estimators that are limited to a linear number of estimates. Our result follows from a general technique that counter-intuitively involves actively perturbing and coordinating the estimates, using techniques developed for privacy preservation. We give additional applications of this technique to our question.Keywords
All Related Versions
Funding Information
- National Science Foundation (CNS 1253345, CCF 1101389)
This publication has 30 references indexed in Scilit:
- Raise standards for preclinical cancer researchNature, 2012
- A firm foundation for private data analysisCommunications of the ACM, 2011
- α-Investing: a Procedure for Sequential Control of Expected False DiscoveriesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimizationAdvances in Computational Mathematics, 2006
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- Contradicted and Initially Stronger Effects in Highly Cited Clinical ResearchJAMA, 2005
- General conditions for predictivity in learning theoryNature, 2004
- 10.1162/153244302760200704Applied Physics Letters, 2000
- Efficient noise-tolerant learning from statistical queriesJournal of the ACM, 1998
- A Note on Screening Regression EquationsThe American Statistician, 1983