Model Averaging for Support Vector Machine by Cross-Validation
Preprint
- 6 January 2023
- preprint
- Published by Springer Nature in Research Square
Abstract
Support vector machine (SVM) is a well-known statistical technique for classification problems in machine learning and other fields.An important question for SVM is the selection of covariates (or features) for the model.Many studies have considered model selection methods. As is well-known, selecting one winning model over others can entail considerable instability in predictive performance due to model selection uncertainties. This paper advocates model averaging as an alternative approach, where estimates obtained from different models are combined in a weighted average. We propose a model weighting scheme and provide the theoretical underpinning for the proposed method. In particular, we prove that our proposed method yields a model average estimator that achieves the smallest hinge risk among all feasible combinations asymptotically. To remedy the computational burden due to a large number of feasible models, we propose a screening step to eliminate the uninformative features before combining the models. Results from real data applications and a simulation study show that the proposed method generally yields more accurate estimates than existing methods.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Optimal Model Averaging Estimation for Generalized Linear Models and Generalized Linear Mixed-Effects ModelsJournal of the American Statistical Association, 2016
- Model Selection via Bayesian Information Criterion for Quantile Regression ModelsJournal of the American Statistical Association, 2014
- Adaptively combined forecasting for discrete response time seriesJournal of Econometrics, 2013
- Oracle properties of SCAD-penalized support vector machineJournal of Statistical Planning and Inference, 2012
- Quantile Regression for Analyzing Heterogeneity in Ultra-High DimensionJournal of the American Statistical Association, 2012
- Jackknife model averagingJournal of Econometrics, 2012
- Least squares model averaging by Mallows criterionJournal of Econometrics, 2010
- Gene Selection for Cancer Classification using Support Vector MachinesMachine Learning, 2002
- A Decision-Theoretic Generalization of On-Line Learning and an Application to BoostingJournal of Computer and System Sciences, 1997
- Analysis of hidden units in a layered network trained to classify sonar targetsNeural Networks, 1988