Bernstein-von Mises for Adaptively Collected Data
Preprint
Abstract
Uncertainty quantification (UQ) for adaptively collected data, such as that coming from adaptive experiments, bandits, or reinforcement learning, is necessary for critical elements of data collection such as ensuring safety and conducting after-study inference. The data's adaptivity creates significant challenges for frequentist UQ, yet Bayesian UQ remains the same as if the data were independent and identically distributed (i.i.d.), making it an appealing and commonly used approach. Bayesian UQ requires the (correct) specification of a prior distribution while frequentist UQ does not, but for i.i.d. data the celebrated Bernstein-von Mises theorem shows that as the sample size grows, the prior 'washes out' and Bayesian UQ becomes frequentist-valid, implying that the choice of prior need not be a major impediment to Bayesian UQ as it makes no difference asymptotically. This paper for the first time extends the Bernstein-von Mises theorem to adaptively collected data, proving asymptotic equivalence between Bayesian UQ and Wald-type frequentist UQ in this challenging setting. Our result showing this asymptotic agreement does not require the standard stability condition required by works studying validity of Wald-type frequentist UQ; in cases where stability is satisfied, our results combined with these prior studies of frequentist UQ imply frequentist validity of Bayesian UQ. Counterintuitively however, they also provide a negative result that Bayesian UQ is not asymptotically frequentist valid when stability fails, despite the fact that the prior washes out and Bayesian UQ asymptotically matches standard Wald-type frequentist UQ. We empirically validate our theory (positive and negative) via a range of simulations.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Reinforcement Learning in Healthcare: A SurveyACM Computing Surveys, 2021
- mHealth app using machine learning to increase physical activity in diabetes and depression: clinical trial protocol for the DIAMANTE StudyBMJ Open, 2020
- Encouraging Physical Activity in Patients With Diabetes: Intervention Using a Reinforcement Learning SystemJournal of Medical Internet Research, 2017
- A Bernstein–von Mises theorem for smooth functionals in semiparametric modelsThe Annals of Statistics, 2015
- The Bernstein–von Mises theorem and nonregular modelsThe Annals of Statistics, 2014
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002
- Converse Results for Existence of Moments and Uniform Integrability for Stopped Random WalksThe Annals of Probability, 1986