The art of BART: Minimax optimality over nonhomogeneous smoothness in high dimension
Preprint
Abstract
Many asymptotically minimax procedures for function estimation often rely on somewhat arbitrary and restrictive assumptions such as isotropy or spatial homogeneity. This work enhances the theoretical understanding of Bayesian additive regression trees under substantially relaxed smoothness assumptions. We provide a comprehensive study of asymptotic optimality and posterior contraction of Bayesian forests when the regression function has anisotropic smoothness that possibly varies over the function domain. The regression function can also be possibly discontinuous. We introduce a new class of sparse {\em piecewise heterogeneous anisotropic} H\"{o}lder functions and derive their minimax lower bound of estimation in high-dimensional scenarios under the $L_2$-loss. We then find that the Bayesian tree priors, coupled with a Dirichlet subset selection prior for sparse estimation in high-dimensional scenarios, adapt to unknown heterogeneous smoothness, discontinuity, and sparsity. These results show that Bayesian forests are uniquely suited for more general estimation problems that would render other default machine learning tools, such as Gaussian processes, suboptimal. Our numerical study shows that Bayesian forests often outperform other competitors such as random forests and deep neural networks, which are believed to work well for discontinuous or complicated smooth functions. Beyond nonparametric regression, we also examined posterior contraction of Bayesian forests for density estimation and binary classification using the technique developed in this study.Keywords
All Related Versions
This publication has 43 references indexed in Scilit:
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spacesNeural Networks, 2020
- Bayesian additive regression trees and the General BART modelStatistics in Medicine, 2019
- Optimal approximation of piecewise smooth functions using deep ReLU neural networksNeural Networks, 2018
- Bayesian Regression Trees for High-Dimensional Prediction and Variable SelectionJournal of the American Statistical Association, 2018
- Nonparametric survival analysis using Bayesian Additive Regression Trees (BART)Statistics in Medicine, 2016
- Multinomial probit Bayesian additive regression treesStat, 2016
- New tight frames of curvelets and optimal representations of objects with piecewise C2 singularitiesCommunications on Pure and Applied Mathematics, 2003
- Risk bounds for model selection via penalizationProbability Theory and Related Fields, 1999
- Rates of convergence for minimum contrast estimatorsProbability Theory and Related Fields, 1993
- On estimating a density using Hellinger distance and some other strange factsProbability Theory and Related Fields, 1986