ROS Regression: Integrating Regularization with Optimal Scaling Regression
Authors:
- Jacqueline J. MeulmanJMJacqueline J. Meulman
- Anita J. van der KooijAKAnita J. van der Kooij
- Kevin L. W. DuistersKDKevin L. W. Duisters
Open Access
- 1 August 2019
- journal article
- research article
- Published by Institute of Mathematical Statistics in Statistical Science
- Vol. 34 (3) , 361-390
Abstract
We present a methodology for multiple regression analysis that deals with categorical variables (possibly mixed with continuous ones), in combination with regularization, variable selection and high-dimensional data ($P\gg N$). Regularization and optimal scaling (OS) are two important extensions of ordinary least squares regression (OLS) that will be combined in this paper. There are two data analytic situations for which optimal scaling was developed. One is the analysis of categorical data, and the other the need for transformations because of nonlinear relationships between predictors and outcome. Optimal scaling of categorical data finds quantifications for the categories, both for the predictors and for the outcome variables, that are optimal for the regression model in the sense that they maximize the multiple correlation. When nonlinear relationships exist, nonlinear transformation of predictors and outcome maximize the multiple correlation in the same way. We will consider a variety of transformation types; typically we use step functions for categorical variables, and smooth (spline) functions for continuous variables. Both types of functions can be restricted to be monotonic, preserving the ordinal information in the data. In combination with optimal scaling, three popular regularization methods will be considered: Ridge regression, the Lasso and the Elastic Net. The resulting method will be called ROS Regression (Regularized Optimal Scaling Regression). The OS algorithm provides straightforward and efficient estimation of the regularized regression coefficients, automatically gives the Group Lasso and Blockwise Sparse Regression, and extends them by the possibility to maintain ordinal properties in the data. Extended examples are provided.Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- Gaussian copula marginal regressionElectronic Journal of Statistics, 2012
- Better Subset Regression Using the Nonnegative GarroteTechnometrics, 1995
- Flexible Discriminant Analysis by Optimal ScoringJournal of the American Statistical Association, 1994
- Estimating Optimal Transformations for Multiple Regression and CorrelationJournal of the American Statistical Association, 1985
- Estimating the Error Rate of a Prediction Rule: Improvement on Cross-ValidationJournal of the American Statistical Association, 1983
- Projection Pursuit RegressionJournal of the American Statistical Association, 1981
- Additive Structure in Qualitative Data: An Alternating Least Squares Method with Optimal Scaling FeaturesPsychometrika, 1976
- Ridge Regression: Applications to Nonorthogonal ProblemsTechnometrics, 1970
- Ridge Regression: Biased Estimation for Nonorthogonal ProblemsTechnometrics, 1970
- Nonmetric Multidimensional Scaling: A Numerical MethodPsychometrika, 1964