Constrained Bayesian Optimization with Noisy Experiments
Authors:
- Benjamin LethamBLBenjamin Letham
- Brian KarrerBKBrian Karrer
- Guilherme OttoniGOGuilherme Ottoni
- Eytan BakshyEBEytan Bakshy
Top Cited Papers
Open Access
- 1 June 2019
- journal article
- research article
- Published by Institute of Mathematical Statistics in Bayesian Analysis
- Vol. 14 (2) , 495-519
Abstract
Randomized experiments are the gold standard for evaluating the effects of changes to real-world systems. Data in these tests may be difficult to collect and outcomes may have high variance, resulting in potentially large measurement error. Bayesian optimization is a promising technique for efficiently optimizing multiple continuous parameters, but existing approaches degrade in performance when the noise level is high, limiting its applicability to many randomized experiments. We derive an expression for expected improvement under greedy batch optimization with noisy observations and noisy constraints, and develop a quasi-Monte Carlo approximation that allows it to be efficiently optimized. Simulations with synthetic functions show that optimization performance on noisy, constrained problems outperforms existing methods. We further demonstrate the effectiveness of the method with two real-world experiments conducted at Facebook: optimizing a ranking system, and optimizing server compiler flags.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- High-dimensional integration: The quasi-Monte Carlo wayActa Numerica, 2013
- Quantile-Based Optimization of Noisy Computer Experiments With Tunable PrecisionTechnometrics, 2012
- Estimating Individualized Treatment Rules Using Outcome Weighted LearningJournal of the American Statistical Association, 2012
- The Correlated Knowledge Gradient for Simulation Optimization of Continuous Parameters using Gaussian Process RegressionSIAM Journal on Optimization, 2011
- Bayesian Guided Pattern Search for Robust Local OptimizationTechnometrics, 2009
- Global optimization of expensive-to-evaluate functions: an empirical comparison of two sampling criteriaJournal of Global Optimization, 2008
- Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-ModelsJournal of Global Optimization, 2006
- The effective dimension and quasi-Monte Carlo integrationJournal of Complexity, 2003
- Scrambling Sobol' and Niederreiter–Xing PointsJournal of Complexity, 1998
- ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLESBiometrika, 1933