A Regularized Newton Method for \({\boldsymbol{\ell}}_{q}\) -Norm Composite Optimization Problems
- 1 August 2023
- journal article
- research article
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Optimization
- Vol. 33 (3) , 1676-1706
Abstract
This paper is concerned with [Formula: see text]-norm regularized minimization problems with a twice continuously differentiable loss function. For this class of nonconvex and nonsmooth composite problems, many algorithms have been proposed to solve them, most of which are of the first-order type. In this work, we propose a hybrid of the proximal gradient method and the subspace regularized Newton method, called HpgSRN. The whole iterate sequence produced by HpgSRN is proved to have a finite length and to converge to an [Formula: see text]-type stationary point under a mild curve-ratio condition and the Kurdyka–Łojasiewicz property of the cost function; it converges linearly if a further Kurdyka–Łojasiewicz property of exponent [Formula: see text] holds. Moreover, a superlinear convergence rate for the iterate sequence is also achieved under an additional local error bound condition. Our convergence results do not require the isolatedness and strict local minimality properties of the [Formula: see text]-stationary point. Numerical comparisons with ZeroFPR, a hybrid of proximal gradient method and quasi-Newton method for the forward-backward envelope of the cost function, proposed in [A. Themelis, L. Stella, and P. Patrinos, SIAM J. Optim., 28 (2018), pp. 2274–2303] for the [Formula: see text]-norm regularized linear and logistic regressions on real data, indicate that HpgSRN not only requires much less computing time but also yields comparable or even better sparsities and objective function values.Keywords
Funding Information
- National Natural Science Foundation of China (1197117)
- Research Grants Council, University Grants Committee (15217520)
This publication has 47 references indexed in Scilit:
- $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast SolverIEEE Transactions on Neural Networks and Learning Systems, 2012
- Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methodsMathematical Programming, 2011
- Lower Bound Theory of Nonzero Entries in Solutions of $\ell_2$-$\ell_p$ MinimizationSIAM Journal on Scientific Computing, 2010
- Efficient Reconstruction of Piecewise Constant Images Using Nonsmooth Nonconvex MinimizationSIAM Journal on Imaging Sciences, 2008
- Local behavior of an iterative framework for generalized equations with nonisolated solutionsMathematical Programming, 2002
- A Krylov--Schur Algorithm for Large EigenproblemsSIAM Journal on Matrix Analysis and Applications, 2002
- A Nonlinear Lagrangian Approach to Constrained Optimization ProblemsSIAM Journal on Optimization, 2001
- Nonlinear ProgrammingJournal of the Operational Research Society, 1997
- A Statistical View of Some Chemometrics Regression ToolsTechnometrics, 1993
- Two-Point Step Size Gradient MethodsIMA Journal of Numerical Analysis, 1988