Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization, II: Shrinking Procedures and Optimal Algorithms
Authors:
- Saeed GhadimiSGSaeed Ghadimi
- Guanghui LanGLGuanghui Lan
Top Cited Papers
- 1 January 2013
- journal article
- research article
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Optimization
- Vol. 23 (4) , 2061-2089
Abstract
In this paper we study new stochastic approximation (SA) type algorithms, namely, the accelerated SA (AC-SA), for solving strongly convex stochastic composite optimization (SCO) problems. Specifically, by introducing a domain shrinking procedure, we significantly improve the large-deviation results associated with the convergence rate of a nearly optimal AC-SA algorithm presented by Ghadimi and Lan in [SIAM J. Optim., 22 (2012), pp 1469--1492]. Moreover, we introduce a multistage AC-SA algorithm, which possesses an optimal rate of convergence for solving strongly convex SCO problems in terms of the dependence on not only the target accuracy, but also a number of problem parameters and the selection of initial points. To the best of our knowledge, this is the first time that such an optimal method has been presented in the literature. From our computational results, these AC-SA algorithms can substantially outperform the classical SA and some other SA type algorithms for solving certain classes of strongly convex SCO problems.Keywords
This publication has 16 references indexed in Scilit:
- Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic FrameworkSIAM Journal on Optimization, 2012
- Validation analysis of mirror descent stochastic approximation methodMathematical Programming, 2011
- An optimal method for stochastic composite optimizationMathematical Programming, 2011
- Robust Stochastic Approximation Approach to Stochastic ProgrammingSIAM Journal on Optimization, 2009
- Interior Gradient and Proximal Methods for Convex and Conic OptimizationSIAM Journal on Optimization, 2006
- Interior projection-like methods for monotone variational inequalitiesMathematical Programming, 2005
- The Sample Average Approximation Method for Stochastic Discrete OptimizationSIAM Journal on Optimization, 2002
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992
- stochastic quasigradient methods and their application to system optimization†Stochastics, 1983
- A Stochastic Approximation MethodThe Annals of Mathematical Statistics, 1951