Node harvest
Author:
Open Access
- 1 December 2010
- journal article
- research article
- Published by Institute of Mathematical Statistics in The Annals of Applied Statistics
- Vol. 4 (4) , 2049-2072
Abstract
When choosing a suitable technique for regression and classification with multivariate predictor variables, one is often faced with a tradeoff between interpretability and high predictive accuracy. To give a classical example, classification and regression trees are easy to understand and interpret. Tree ensembles like Random Forests provide usually more accurate predictions. Yet tree ensembles are also more difficult to analyze than single trees and are often criticized, perhaps unfairly, as ‘black box’ predictors. Node harvest is trying to reconcile the two aims of interpretability and predictive accuracy by combining positive aspects of trees and tree ensembles. Results are very sparse and interpretable and predictive accuracy is extremely competitive, especially for low signal-to-noise data. The procedure is simple: an initial set of a few thousand nodes is generated randomly. If a new observation falls into just a single node, its prediction is the mean response of all training observation within this node, identical to a tree-like prediction. A new observation falls typically into several nodes and its prediction is then the weighted average of the mean responses across all these nodes. The only role of node harvest is to ‘pick’ the right nodes from the initial large ensemble of nodes by choosing node weights, which amounts in the proposed algorithm to a quadratic programming problem with linear inequality constraints. The solution is sparse in the sense that only very few nodes are selected with a nonzero weight. This sparsity is not explicitly enforced. Maybe surprisingly, it is not necessary to select a tuning parameter for optimal predictive accuracy. Node harvest can handle mixed data and missing values and is shown to be simple to interpret and competitive in predictive accuracy on a variety of data sets.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Forest GarroteElectronic Journal of Statistics, 2009
- Random Forests and Adaptive Nearest NeighborsJournal of the American Statistical Association, 2006
- Probabilistic Sensitivity Analysis of Complex Models: A Bayesian ApproachJournal of the Royal Statistical Society Series B: Statistical Methodology, 2004
- Least angle regressionThe Annals of Statistics, 2004
- Atomic Decomposition by Basis PursuitSIAM Review, 2001
- Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)The Annals of Statistics, 2000
- Do-it-yourself climate predictionNature, 1999
- Shape Quantization and Recognition with Randomized TreesNeural Computation, 1997
- Bagging predictorsMachine Learning, 1996
- Stacked generalizationNeural Networks, 1992