Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression
Authors:
Open Access
- 14 August 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1510-1520
Abstract
Gradient Boosting Machines (GBM) are hugely popular for solving tabular data problems. However, practitioners are not only interested in point predictions, but also in probabilistic predictions in order to quantify the uncertainty of the predictions. Creating such probabilistic predictions is difficult with existing GBM-based solutions: they either require training multiple models or they become too computationally expensive to be useful for large-scale settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of decision trees in a computationally efficient manner. PGBM approximates the leaf weights in a decision tree as a random variable, and approximates the mean and variance of each sample in a dataset via stochastic tree ensemble update equations. These learned moments allow us to subsequently sample from a specified distribution after training. We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods: (i) PGBM enables probabilistic estimates without compromising on point performance in a single model, (ii) PGBM learns probabilistic estimates via a single model only (and without requiring multi-parameter boosting), and thereby offers a speedup of up to several orders of magnitude over existing state-of-the-art methods on large datasets, and (iii) PGBM achieves accurate probabilistic estimates in tasks with complex differentiable loss functions, such as hierarchical time series problems, where we observed up to 10% improvement in point forecasting performance and up to 300% improvement in probabilistic forecasting performance.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- DeepAR: Probabilistic forecasting with autoregressive recurrent networksInternational Journal of Forecasting, 2020
- Forecasting at ScaleThe American Statistician, 2018
- Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather ForecastsMathematical Geosciences, 2017
- Probabilistic demand forecasting at scaleProceedings of the VLDB Endowment, 2017
- XGBoost2016
- Generalized Additive Models for Location, Scale and ShapeJournal of the Royal Statistical Society Series C: Applied Statistics, 2005
- Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 2001
- Distribution of Residual Autocorrelations in Autoregressive-Integrated Moving Average Time Series ModelsJournal of the American Statistical Association, 1970