Distributed stochastic optimization and learning
Authors:
- 1 September 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 850-857
Abstract
We consider the problem of distributed stochastic optimization, where each of several machines has access to samples from the same source distribution, and the goal is to jointly optimize the expected objective w.r.t. the source distribution, minimizing: (1) overall runtime; (2) communication costs; (3) number of samples used. We study this problem systematically, highlighting fundamental limitations, and differences versus distributed consensus problems where each machine has a different, independent, objective. We show how the best known guarantees are obtained by an accelerated mini-batched SGD approach, and contrast the runtime and sample costs of the approach with those of other distributed optimization algorithms.Keywords
This publication has 8 references indexed in Scilit:
- Distributed Optimization and Statistical Learning via the Alternating Direction Method of MultipliersFoundations and Trends in Machine Learning, 2011
- An optimal method for stochastic composite optimizationMathematical Programming, 2011
- Pegasos: primal estimated sub-gradient solver for SVMMathematical Programming, 2010
- Distributed Stochastic Subgradient Projection Algorithms for Convex OptimizationJournal of Optimization Theory and Applications, 2010
- Robust Stochastic Approximation Approach to Stochastic ProgrammingSIAM Journal on Optimization, 2009
- SVM optimization2008
- Logarithmic regret algorithms for online convex optimizationMachine Learning, 2007
- The Nature of Statistical Learning Theory1995