Revisiting Landscape Analysis in Deep Neural Networks: Eliminating Decreasing Paths to Infinity
Preprint
Abstract
Traditional landscape analysis of deep neural networks aims to show that no sub-optimal local minima exist in some appropriate sense. From this, one may be tempted to conclude that descent algorithms which escape saddle points will reach a good local minimum. However, basic optimization theory tell us that it is also possible for a descent algorithm to diverge to infinity if there are paths leading to infinity, along which the loss function decreases. It is not clear whether for non-linear neural networks there exists one setting that no bad local-min and no decreasing paths to infinity can be simultaneously achieved. In this paper, we give the first positive answer to this question. More specifically, for a large class of over-parameterized deep neural networks with appropriate regularizers, the loss function has no bad local minima and no decreasing paths to infinity. The key mathematical trick is to show that the set of regularizers which may be undesirable can be viewed as the image of a Lipschitz continuous mapping from a lower-dimensional Euclidean space to a higher-dimensional Euclidean space, and thus has zero measure.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural NetworksIEEE Journal on Selected Areas in Information Theory, 2020
- Gradient descent optimizes over-parameterized deep ReLU networksMachine Learning, 2019
- Gradient descent with random initialization: fast global convergence for nonconvex phase retrievalMathematical Programming, 2019
- A mean field view of the landscape of two-layer neural networksProceedings of the National Academy of Sciences, 2018
- Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural NetworksIEEE Transactions on Information Theory, 2018
- A Geometric Analysis of Phase RetrievalFoundations of Computational Mathematics, 2017
- Guaranteed Matrix Completion via Non-Convex FactorizationIEEE Transactions on Information Theory, 2016
- Deep Residual Learning for Image Recognition2016
- Interference Alignment Using Finite and Dependent Channel Extensions: The Single Beam CaseIEEE Transactions on Information Theory, 2014
- Convergence of a Block Coordinate Descent Method for Nondifferentiable MinimizationJournal of Optimization Theory and Applications, 2001