Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk
Authors:
Preprint
Abstract
We examine the theoretical properties of enforcing priors provided by generative deep neural networks via empirical risk minimization. In particular we consider two models, one in which the task is to invert a generative neural network given access to its last layer and another in which the task is to invert a generative neural network given only compressive linear observations of its last layer. We establish that in both cases, in suitable regimes of network layer sizes and a randomness assumption on the network weights, that the non-convex objective function given by empirical risk minimization does not have any spurious stationary points. That is, we establish that with high probability, at any point away from small neighborhoods around two scalar multiples of the desired solution, there is a descent direction. Hence, there are no local minima, saddle points, or other stationary points outside these neighborhoods. These results constitute the first theoretical guarantees which establish the favorable global geometry of these non-convex optimization problems, and they bridge the gap between the empirical success of enforcing deep generative priors and a rigorous understanding of non-linear inverse problems.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Rapid, robust, and reliable blind deconvolution via nonconvex optimizationApplied and Computational Harmonic Analysis, 2019
- Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind DeconvolutionFoundations of Computational Mathematics, 2019
- The Projected Power Method: An Efficient Algorithm for Joint Alignment from Pairwise DifferencesCommunications on Pure and Applied Mathematics, 2018
- A Geometric Analysis of Phase RetrievalFoundations of Computational Mathematics, 2017
- Group Invariant ScatteringCommunications on Pure and Applied Mathematics, 2012
- Counting faces of randomly projected polytopes when the projection radically lowers dimensionJournal of the American Mathematical Society, 2008
- A Simple Proof of the Restricted Isometry Property for Random MatricesConstructive Approximation, 2008
- Sparse MRI: The application of compressed sensing for rapid MR imagingMagnetic Resonance in Medicine, 2007
- For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solutionCommunications on Pure and Applied Mathematics, 2006
- Stable signal recovery from incomplete and inaccurate measurementsCommunications on Pure and Applied Mathematics, 2006