Structured variable selection in support vector machines
Authors:
- Seongho WuSWSeongho Wu
- Hui ZouHZHui Zou
- Ming YuanMYMing Yuan
Open Access
- 1 January 2008
- journal article
- research article
- Published by Institute of Mathematical Statistics in Electronic Journal of Statistics
- Vol. 2 (none) , 103-117
Abstract
When applying the support vector machine (SVM) to high-dimensional classification problems, we often impose a sparse structure in the SVM to eliminate the influences of the irrelevant predictors. The lasso and other variable selection techniques have been successfully used in the SVM to perform automatic variable selection. In some problems, there is a natural hierarchical structure among the variables. Thus, in order to have an interpretable SVM classifier, it is important to respect the heredity principle when enforcing the sparsity in the SVM. Many variable selection methods, however, do not respect the heredity principle. In this paper we enforce both sparsity and the heredity principle in the SVM by using the so-called structured variable selection (SVS) framework originally proposed in [20]. We minimize the empirical hinge loss under a set of linear inequality constraints and a lasso-type penalty. The solution always obeys the desired heredity principle and enjoys sparsity. The new SVM classifier can be efficiently fitted, because the optimization problem is a linear program. Another contribution of this work is to present a nonparametric extension of the SVS framework, and we propose nonparametric heredity SVMs. Simulated and real data are used to illustrate the merits of the proposed method.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Structured variable selection and estimationThe Annals of Applied Statistics, 2009
- On the Non-Negative Garrotte EstimatorJournal of the Royal Statistical Society Series B: Statistical Methodology, 2007
- Model Selection and Estimation in Regression with Grouped VariablesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2005
- Efficient Empirical Bayes Variable Selection and Estimation in Linear ModelsJournal of the American Statistical Association, 2005
- Efficient quadratic regularization for expression arraysBiostatistics, 2004
- Least angle regressionThe Annals of Statistics, 2004
- The Elements of Statistical Learning2001
- A Bayesian Variable-Selection Approach for Analyzing Designed Experiments With Complex AliasingTechnometrics, 1997
- Bayesian variable selection with related predictorsCanadian Journal of Statistics, 1996
- Better Subset Regression Using the Nonnegative GarroteTechnometrics, 1995