Preserving Statistical Validity in Adaptive Data Analysis
Authors:
Preprint
Abstract
A great deal of effort has been devoted to reducing the risk of spurious scientific discoveries, from the use of sophisticated validation techniques, to deep statistical methods for controlling the false discovery rate in multiple hypothesis testing. However, there is a fundamental disconnect between the theoretical results and the practice of data analysis: the theory of statistical inference assumes a fixed collection of hypotheses to be tested, or learning algorithms to be applied, selected non-adaptively before the data are gathered, whereas in practice data is shared and reused with hypotheses and new analyses being generated on the basis of data exploration and the outcomes of previous analyses. In this work we initiate a principled study of how to guarantee the validity of statistical inference in adaptive data analysis. As an instance of this problem, we propose and investigate the question of estimating the expectations of $m$ adaptively chosen functions on an unknown distribution given $n$ random samples. We show that, surprisingly, there is a way to estimate an exponential in $n$ number of expectations accurately even if the functions are chosen adaptively. This gives an exponential improvement over standard empirical estimators that are limited to a linear number of estimates. Our result follows from a general technique that counter-intuitively involves actively perturbing and coordinating the estimates, using techniques developed for privacy preservation. We give additional applications of this technique to our question.
Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Interactive fingerprinting codes and the hardness of preventing false discovery2016
- Preventing False Discovery in Interactive Data Analysis Is Hard2014
- Believe it or not: how much can we rely on published data on potential drug targets?Nature Reviews Drug Discovery, 2011
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant2011
- The Quality Preserving Database: A Computational Framework for Encouraging Collaboration, Enhancing Power and Controlling False DiscoveryIEEE/ACM Transactions on Computational Biology and Bioinformatics, 2010
- α-Investing: a Procedure for Sequential Control of Expected False DiscoveriesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2008
- Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimizationAdvances in Computational Mathematics, 2006
- Contradicted and Initially Stronger Effects in Highly Cited Clinical ResearchJAMA, 2005
- General conditions for predictivity in learning theoryNature, 2004
- Privacy-Preserving Datamining on Vertically Partitioned Databases2004