Tabular Data: Deep Learning is Not All You Need
Preprint
Abstract
A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Theoretical issues in deep networksProceedings of the National Academy of Sciences, 2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Deep Neural Decision Forests2015
- Hyperopt: a Python library for model selection and hyperparameter optimizationComputational Science & Discovery, 2015
- The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of VarianceJournal of the American Statistical Association, 1937