On the selection of decision trees in Random Forests
Authors:
- 1 June 2009
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 21614393,p. 302-307
Abstract
In this paper we present a study on the random forest (RF) family of ensemble methods. In a ldquoclassicalrdquo RF induction process a fixed number of randomized decision trees are inducted to form an ensemble. This kind of algorithm presents two main drawbacks : (i) the number of trees has to be fixed a priori (ii) the interpretability and analysis capacities offered by decision tree classifiers are lost due to the randomization principle. This kind of process in which trees are independently added to the ensemble, offers no guarantee that all those trees will cooperate effectively in the same committee. This statement rises two questions: are there any decision trees in a RF that provide the deterioration of ensemble performance? If so, is it possible to form a more accurate committee via removal of decision trees with poor performance? The answer to these questions is tackled as a classifier selection problem. We thus show that better subsets of decision trees can be obtained even using a sub-optimal classifier selection method. This proves that ldquoclassicalrdquo RF induction process, for which randomized trees are arbitrary added to the ensemble, is not the best approach to produce accurate RF classifiers. We also show the interest in designing RF by adding trees in a more dependent way than it is traditionally done in ldquoclassicalrdquo RF induction algorithms.Keywords
This publication has 10 references indexed in Scilit:
- Rotation Forest: A New Classifier Ensemble MethodIEEE Transactions on Pattern Analysis and Machine Intelligence, 2006
- Fast branch & bound algorithms for optimal feature selectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2004
- Random ForestsMachine Learning, 2001
- Methods for Designing Multiple Classifier Systems2001
- Limiting the Number of Trees in Random Forests2001
- Arcing classifier (with discussion and a rejoinder by the author)The Annals of Statistics, 1998
- The random subspace method for constructing decision forestsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Wrappers for feature subset selectionArtificial Intelligence, 1997
- Bagging predictorsMachine Learning, 1996