Online Statistical Inference for Parameters Estimation with Linear-Equality Constraints
Authors:
Preprint
Abstract
Stochastic gradient descent (SGD) and projected stochastic gradient descent (PSGD) are scalable algorithms to compute model parameters in unconstrained and constrained optimization problems. In comparison with SGD, PSGD forces its iterative values into the constrained parameter space via projection. From a statistical point of view, this paper studies the limiting distribution of PSGD-based estimate when the true parameters satisfy some linear-equality constraints. Our theoretical findings reveal the role of projection played in the uncertainty of the PSGD-based estimate. As a byproduct, we propose an online hypothesis testing procedure to test the linear-equality constraints. Simulation studies on synthetic data and an application to a real-world dataset confirm our theory.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Optimization Methods for Large-Scale Machine LearningSIAM Review, 2018
- An averaged projected Robbins-Monro algorithm for estimating the parameters of a truncated spherical distributionElectronic Journal of Statistics, 2017
- A useful variant of the Davis–Kahan theorem for statisticiansBiometrika, 2014
- Large-scale matrix factorization with distributed stochastic gradient descent2011
- Robust Stochastic Approximation Approach to Stochastic ProgrammingSIAM Journal on Optimization, 2009
- Maximum-Likelihood Estimation, the CramÉr–Rao Bound, and the Method of Scoring With Parameter ConstraintsIEEE Transactions on Signal Processing, 2008
- Asymptotic Almost Sure Efficiency of Averaged Stochastic AlgorithmsSIAM Journal on Control and Optimization, 2000
- Acceleration of Stochastic Approximation by AveragingSIAM Journal on Control and Optimization, 1992
- Lower bounds for parametric estimation with constraintsIEEE Transactions on Information Theory, 1990
- Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung)Mathematische Annalen, 1912