Convergence beyond the over-parameterized regime using Rayleigh quotients
Preprint
Abstract
In this paper, we present a new strategy to prove the convergence of deep learning architectures to a zero training (or even testing) loss by gradient flow. Our analysis is centered on the notion of Rayleigh quotients in order to prove Kurdyka-Łojasiewicz inequalities for a broader set of neural network architectures and loss functions. We show that Rayleigh quotients provide a unified view for several convergence analysis techniques in the literature. Our strategy produces a proof of convergence for various examples of parametric learning. In particular, our analysis does not require the number of parameters to tend to infinity, nor the number of samples to be finite, thus extending to test loss minimization and beyond the over-parameterized regime.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Loss landscapes and optimization in over-parameterized non-linear systems and neural networksApplied and Computational Harmonic Analysis, 2022
- Sparse optimization on measures with over-parameterized gradient descentMathematical Programming, 2021
- Gradient descent optimizes over-parameterized deep ReLU networksMachine Learning, 2019
- The Łojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical SystemsSIAM Journal on Optimization, 2007
- On gradients of functions definable in o-minimal structuresAnnales de l'institut Fourier, 1998
- On the LambertW functionAdvances in Computational Mathematics, 1996
- Universal approximation bounds for superpositions of a sigmoidal functionIEEE Transactions on Information Theory, 1993
- Multilayer feedforward networks with a nonpolynomial activation function can approximate any functionNeural Networks, 1993
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989