Normalization effects on shallow neural networks and related asymptotic expansions
Preprint
Abstract
We consider shallow (single hidden layer) neural networks and characterize their performance when trained with stochastic gradient descent as the number of hidden units $N$ and gradient descent steps grow to infinity. In particular, we investigate the effect of different scaling schemes, which lead to different normalizations of the neural network, on the network's statistical output, closing the gap between the $1/\sqrt{N}$ and the mean-field $1/N$ normalization. We develop an asymptotic expansion for the neural network's statistical output pointwise with respect to the scaling parameter as the number of hidden units grows to infinity. Based on this expansion, we demonstrate mathematically that to leading order in $N$, there is no bias-variance trade off, in that both bias and variance (both explicitly characterized) decrease as the number of hidden units increases and time grows. In addition, we show that to leading order in $N$, the variance of the neural network's statistical output decays as the implied normalization by the scaling parameter approaches the mean field normalization. Numerical studies on the MNIST and CIFAR10 datasets show that test and train accuracy monotonically improve as the neural network's normalization gets closer to the mean field normalization.
Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Gradient descent optimizes over-parameterized deep ReLU networksMachine Learning, 2019
- DGM: A deep learning algorithm for solving partial differential equationsJournal of Computational Physics, 2018
- Dermatologist-level classification of skin cancer with deep neural networksNature, 2017
- Reynolds averaged turbulence modelling using deep neural networks with embedded invarianceJournal of Fluid Mechanics, 2016
- Machine learning strategies for systems with invariance propertiesJournal of Computational Physics, 2016
- Stochastic Approximation and Recursive Algorithms and Applications2003
- Large deviations and mean-field theory for asymmetric random recurrent neural networksProbability Theory and Related Fields, 2002
- Nonlinearity creates linear independenceAdvances in Computational Mathematics, 1996
- Approximation capabilities of multilayer feedforward networksNeural Networks, 1991
- Multilayer feedforward networks are universal approximatorsNeural Networks, 1989