CoffeeBoost: Gradient Boosting Native Conformal Inference for Bayesian Optimization
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (17) , 18017-18025
Abstract
Bayesian optimization (BO) is a key technique for solving black-box optimization problems. This study extends the scope of BO from conventional applications (e.g., AutoML and robotics learning) to automated tuning of software systems. Despite GP (Gaussian Process) implementing a foundation formalism for exploitation and exploration in BO, its limited predictive power and unrealistic assumptions (e.g., continuity and Gaussianity) can severely affect its effectiveness and efficiency in tuning complex software systems. To overcome these limitations, we propose a BO framework CoffeeBoost, which implements exploitation and exploration with a GBDT-native distribution-free probabilistic surrogate model. CoffeeBoost constructs surrogate models via stochastic gradient boosting ensembles (SGBE) and quantifies probabilistic distributions via distribution-free conformal predictive systems. Moreover, CoffeeBoost leverages the residual paths in SGBE to improve the local adaptiveness of the resulting predictive distributions in a GBDT-native manner. Across eight auto-tuning benchmarks for database management systems (DBMS), we evaluate CoffeeBoost and show its superior learnability and optimizability against existing GP-based and tree-ensemble-based BO schemes. Detailed analysis further shows CoffeeBoost's predictive distributions excel in both coverage and tightness.Keywords
This publication has 11 references indexed in Scilit:
- Recent Advances in Bayesian OptimizationACM Computing Surveys, 2023
- Conformal Predictive Distribution TreesAnnals of Mathematics and Artificial Intelligence, 2023
- Bayesian optimization for chemical products and functional materialsCurrent Opinion in Chemical Engineering, 2022
- Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees2022
- Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces2022
- Distribution-Free Predictive Inference for RegressionJournal of the American Statistical Association, 2018
- Accelerating difficulty estimation for conformal regression forestsAnnals of Mathematics and Artificial Intelligence, 2017
- Bayesian Optimization in a Billion Dimensions via Random EmbeddingsJournal of Artificial Intelligence Research, 2016
- Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 2001
- Efficient Global Optimization of Expensive Black-Box FunctionsJournal of Global Optimization, 1998