An Experimental Evaluation of Large Scale GBDT Systems
Preprint
Abstract
Gradient boosting decision tree (GBDT) is a widely-used machine learning algorithm in both data analytic competitions and real-world industrial applications. Further, driven by the rapid increase in data volume, efforts have been made to train GBDT in a distributed setting to support large-scale workloads. However, we find it surprising that the existing systems manage the training dataset in different ways, but none of them have studied the impact of data management. To that end, this paper aims to study the pros and cons of different data management methods regarding the performance of distributed GBDT. We first introduce a quadrant categorization of data management policies based on data partitioning and data storage. Then we conduct an in-depth systematic analysis and summarize the advantageous scenarios of the quadrants. Based on the analysis, we further propose a novel distributed GBDT system named Vero, which adopts the unexplored composition of vertical partitioning and row-store and suits for many large-scale cases. To validate our analysis empirically, we implement different quadrants in the same code base and compare them under extensive workloads, and finally compare Vero with other state-of-the-art systems over a wide range of datasets. Our theoretical and experimental results provide a guideline on choosing a proper data management policy for a given workload.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- An experimental evaluation of large scale GBDT systemsProceedings of the VLDB Endowment, 2019
- Approximate Query Processing: What is New and Where to Go?Data Science and Engineering, 2018
- Moment-based quantile sketches for efficient high cardinality aggregation queriesProceedings of the VLDB Endowment, 2018
- Approximate Calculation of Window Aggregate Functions via Global Random SampleData Science and Engineering, 2018
- Angel: a new large-scale machine learning systemNational Science Review, 2017
- Further results on the margin explanation of boosting: new algorithm and experimentsScience China Information Sciences, 2012
- Exploring Correlated Subspaces for Efficient Query Processing in Sparse DatabasesIEEE Transactions on Knowledge and Data Engineering, 2009
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 2001
- A Decision-Theoretic Generalization of On-Line Learning and an Application to BoostingJournal of Computer and System Sciences, 1997