Shrinkage estimator generalizations of Proximal Support Vector Machines
Author:
Open Access
- 23 July 2002
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 173-182
Abstract
We give a statistical interpretation of Proximal Support Vector Machines (PSVM) proposed at KDD2001 as linear approximaters to (nonlinear) Support Vector Machines (SVM). We prove that PSVM using a linear kernel is identical to ridge regression, a biased-regression method known in the statistical community for more than thirty years. Techniques from the statistical literature to estimate the tuning constant that appears in the SVM and PSVM framework are discussed. Better shrinkage strategies that incorporate more than one tuning constant are suggested. For nonlinear kernels, the minimization problem posed in the PSVM framework is equivalent to finding the posterior mode of a Bayesian model defined through a Gaussian process on the predictor space. Apart from providing new insights, these interpretations help us attach an estimate of uncertainty to our predictions and enable us to build richer classes of models. In particular, we propose a new algorithm called PSVMMIX which is a combination of ridge regression and a Gaussian process model. Extension to the case of continuous response is straightforward and illustrated with example datasets.Keywords
This publication has 20 references indexed in Scilit:
- Bayesian Analysis of Binary and Polychotomous Response DataJournal of the American Statistical Association, 1993
- On ordinary ridge regression in generalized linear modelsCommunications in Statistics - Theory and Methods, 1992
- Sampling-Based Approaches to Calculating Marginal DensitiesJournal of the American Statistical Association, 1990
- Collinearity in generalized linear modelsCommunications in Statistics - Theory and Methods, 1989
- Ridge estimation in logistic regressionCommunications in Statistics - Simulation and Computation, 1988
- A Simulation Study of Some Ridge EstimatorsJournal of the American Statistical Association, 1981
- A Critique of Some Ridge Regression MethodsJournal of the American Statistical Association, 1980
- A Simulation Study of Alternatives to Ordinary Least SquaresJournal of the American Statistical Association, 1977
- Data Analysis Using Stein's Estimator and its GeneralizationsJournal of the American Statistical Association, 1975
- Ridge Regression: Biased Estimation for Nonorthogonal ProblemsTechnometrics, 1970