Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk
- 15 August 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 66 (1) , 401-418
Abstract
We examine the theoretical properties of enforcing priors provided by generative deep neural networks via empirical risk minimization. In particular we consider two models, one in which the task is to invert a generative neural network given access to its last layer and another in which the task is to invert a generative neural network given only compressive linear observations of its last layer. We establish that in both cases, in suitable regimes of network layer sizes and a randomness assumption on the network weights, that the non-convex objective function given by empirical risk minimization does not have any spurious stationary points. That is, we establish that with high probability, at any point away from small neighborhoods around two scalar multiples of the desired solution, there is a descent direction. Hence, there are no local minima, saddle points, or other stationary points outside these neighborhoods. These results constitute the first theoretical guarantees which establish the favorable global geometry of these non-convex optimization problems, and they bridge the gap between the empirical success of enforcing deep generative priors and a rigorous understanding of non-linear inverse problems.Keywords
All Related Versions
Funding Information
- National Science Foundation (DMS-1464525, CAREER DMS-1848087)
This publication has 30 references indexed in Scilit:
- PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex ProgrammingCommunications on Pure and Applied Mathematics, 2012
- Group Invariant ScatteringCommunications on Pure and Applied Mathematics, 2012
- Introduction to the non-asymptotic analysis of random matrices2012
- PDBe: Protein Data Bank in EuropeNucleic Acids Research, 2009
- Exact Matrix Completion via Convex OptimizationFoundations of Computational Mathematics, 2009
- Counting faces of randomly projected polytopes when the projection radically lowers dimensionJournal of the American Mathematical Society, 2008
- Sparse MRI: The application of compressed sensing for rapid MR imagingMagnetic Resonance in Medicine, 2007
- For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solutionCommunications on Pure and Applied Mathematics, 2006
- Stable signal recovery from incomplete and inaccurate measurementsCommunications on Pure and Applied Mathematics, 2006
- Some NP-complete problems in quadratic and nonlinear programmingMathematical Programming, 1987