BoostTree and BoostForest for Ensemble Learning
Authors:
- Changming ZhaoCZChangming Zhao
- Dongrui WuDWDongrui Wu
- Jian HuangJHJian Huang
- Ye YuanYYYe Yuan
- Hai-Tao ZhangHZHai-Tao Zhang
- Ruimin PengRPRuimin Peng
- Zhenhua ShiZSZhenhua Shi
Preprint
Abstract
Bootstrap aggregating (Bagging) and boosting are two popular ensemble learning approaches, which combine multiple base learners to generate a composite model for more accurate and more reliable performance. They have been widely used in biology, engineering, healthcare, etc. This paper proposes BoostForest, which is an ensemble learning approach using BoostTree as base learners and can be used for both classification and regression. BoostTree constructs a tree model by gradient boosting. It increases the randomness (diversity) by drawing the cut-points randomly at node splitting. BoostForest further increases the randomness by bootstrapping the training data in constructing different BoostTrees. BoostForest generally outperformed four classical ensemble learning approaches (Random Forest, Extra-Trees, XGBoost and LightGBM) on 35 classification and regression datasets. Remarkably, BoostForest tunes its parameters by simply sampling them randomly from a parameter pool, which can be easily specified, and its ensemble learning framework can also be used to combine many other base learners.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Lunar impact crater identification and age estimation with Chang’E data by deep and transfer learningNature Communications, 2020
- Improving the informativeness of Mendelian disease-derived pathogenicity scores for common diseaseNature Communications, 2020
- Ensemble dimensionality reduction and feature gene extraction for single-cell RNA-seq dataNature Communications, 2020
- Machine learning can identify newly diagnosed patients with CLL at high risk of infectionNature Communications, 2020
- Massive computational acceleration by using neural networks to emulate mechanism-based biological modelsNature Communications, 2019
- Towards Safe Weakly Supervised LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Pedestrian Detection with Spatially Pooled Features and Structured Ensemble LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Extreme learning machine: Theory and applicationsNeurocomputing, 2006
- Stochastic gradient boostingComputational Statistics & Data Analysis, 2002
- Constructing optimal binary decision trees is NP-completeInformation Processing Letters, 1976