Complexity control by gradient descent in deep networks
Authors:
Open Access
- 24 February 2020
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 11 (1) , 1-5
Abstract
Overparametrized deep networks predict well, despite the lack of an explicit complexity control during training, such as an explicit regularization term. For exponential-type loss functions, we solve this puzzle by showing an effective regularization effect of gradient descent in terms of the normalized weights that are relevant for classification.Keywords
Funding Information
- NSF | NSF Office of the Director | Office of International Science and Engineering (CCF-1231216, CCF-1231216, CCF-1231216)
- JUMP Semiconductor Research Corporation (SRC) program sponsored by DARPA
- NSF | NSF Office of the Director | Office of International Science and Engineering
- NSF | NSF Office of the Director | Office of International Science and Engineering
This publication has 3 references indexed in Scilit:
- Just interpolate: Kernel “Ridgeless” regression can generalizeThe Annals of Statistics, 2020
- On gradient adaptation with unit-norm constraintsIEEE Transactions on Signal Processing, 2000
- Regularization Operators2000