Upper and lower bounds for the Lipschitz constant of random neural networks
- 26 March 2025
- journal article
- Published by Oxford University Press (OUP) in Information and Inference: A Journal of the IMA
- Vol. 14 (2)
Abstract
Empirical studies have widely demonstrated that neural networks are highly sensitive to small, adversarial perturbations of the input. The worst-case robustness against these so-called adversarial examples can be quantified by the Lipschitz constant of the neural network. In this paper, we study upper and lower bounds for the Lipschitz constant of random ReLU neural networks. Specifically, we assume that the weights and biases follow a generalization of the He initialization, where general symmetric distributions for the biases are permitted. For deep networks of fixed depth and sufficiently large width, our established upper bound is larger than the lower bound by a factor that is logarithmic in the width. In contrast, for shallow neural networks we characterize the Lipschitz constant up to an absolute numerical constant that is independent of all parameters.Keywords
All Related Versions
Funding Information
- German Science Foundation (VO 2594/1-1)
- Hightech Agenda Bavaria
This publication has 13 references indexed in Scilit:
- Products of Many Large Random Matrices and Gradients in Deep Neural NetworksCommunications in Mathematical Physics, 2019
- Towards a regularity theory for ReLU networks – chain rule and global error estimates2019
- High-Dimensional Probability2018
- Adversarial Examples in the Physical World2018
- Audio Adversarial Examples: Targeted Attacks on Speech-to-Text2018
- The Limitations of Deep Learning in Adversarial Settings2016
- Probability in High Dimension2014
- Understanding Machine Learning2014
- Measure Theory2013
- Real Analysis and Probability2002