BoostTree and BoostForest for Ensemble Learning
Authors:
Preprint
Abstract
Bootstrap aggregating (Bagging) and boosting are two popular ensemble learning approaches, which combine multiple base learners to generate a composite model for more accurate and more reliable performance. They have been widely used in biology, engineering, healthcare, etc. This article proposes BoostForest, which is an ensemble learning approach using BoostTree as base learners and can be used for both classification and regression. BoostTree constructs a tree model by gradient boosting. It achieves high randomness (diversity) by sampling its parameters randomly from a parameter pool, and selecting a subset of features randomly at node splitting. BoostForest further increases the randomness by bootstrapping the training data in constructing different BoostTrees. BoostForest outperformed four classical ensemble learning approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 34 classification and regression datasets. Remarkably, BoostForest has only one hyper-parameter (the number of BoostTrees), which can be easily specified. Our code is publicly available, and the proposed ensemble learning framework can also be used to combine many other base learners.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Infrared spectroscopy data- and physics-driven machine learning for characterizing surface microstructure of complex materialsNature Communications, 2020
- Learning algorithms allow for improved reliability and accuracy of global mean surface temperature projectionsNature Communications, 2020
- Machine learning can identify newly diagnosed patients with CLL at high risk of infectionNature Communications, 2020
- RNA secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learningNature Communications, 2019
- Massive computational acceleration by using neural networks to emulate mechanism-based biological modelsNature Communications, 2019
- Extreme learning machine: Theory and applicationsNeurocomputing, 2006
- Logistic Model TreesMachine Learning, 2005
- Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble AccuracyMachine Learning, 2003
- Stochastic gradient boostingComputational Statistics & Data Analysis, 2002
- MultiBoosting: A Technique for Combining Boosting and WaggingMachine Learning, 2000