Convergence Analysis of Generalized Iteratively Reweighted Least Squares Algorithms on Convex Function Spaces
Authors:
- Nicolai BissantzNBNicolai Bissantz
- Lutz DümbgenLDLutz Dümbgen
- Axel MunkAMAxel Munk
- Bernd StratmannBSBernd Stratmann
- 1 January 2009
- journal article
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Optimization
- Vol. 19 (4) , 1828-1845
Abstract
The computation of robust regression estimates often relies on minimization of a convex functional on a convex set. In this paper we discuss a general technique for a large class of convex functionals to compute the minimizers iteratively, which is closely related to majorization-minimization algorithms. Our approach is based on a quadratic approximation of the functional to be minimized and includes the iteratively reweighted least squares algorithm as a special case. We prove convergence on convex function spaces for general coercive and convex functionals F and derive geometric convergence in certain unconstrained settings. The algorithm is applied to total variation (TV) penalized quantile regression and is compared with a step size corrected Newton–Raphson algorithm. It is found that typically in the first steps the iteratively reweighted least squares algorithm performs significantly better, whereas the Newton type method outpaces the former only after many iterations. Finally, in the setting of bivariate regression with unimodality constraints we illustrate how this algorithm allows one to utilize highly efficient algorithms for special quadratic programs in more complex settings.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- A General Projection Framework for Constrained SmoothingStatistical Science, 2001
- Optimization Transfer Using Surrogate Objective FunctionsJournal of Computational and Graphical Statistics, 2000
- Local asymptotics for quantile smoothing splinesThe Annals of Statistics, 1997
- Quantile smoothing splinesBiometrika, 1994
- Bootstrap Simultaneous Error Bars for Nonparametric RegressionThe Annals of Statistics, 1991
- Spatial median and directional dataBiometrika, 1987
- A System of Subroutines for Iteratively Reweighted Least Squares ComputationsACM Transactions on Mathematical Software, 1980
- Regression QuantilesEconometrica, 1978
- On Estimating Monotone ParametersThe Annals of Mathematical Statistics, 1968
- Robust Estimation of a Location ParameterThe Annals of Mathematical Statistics, 1964