Near-optimal estimates for the $\ell^p$-Lipschitz constants of deep random ReLU neural networks
Preprint
Abstract
This paper studies the $\ell^p$-Lipschitz constants of ReLU neural networks $Φ: \mathbb{R}^d \to \mathbb{R}$ with random parameters for $p \in [1,\infty]$. The distribution of the weights follows a variant of the He initialization and the biases are drawn from symmetric distributions. We derive high probability upper and lower bounds for wide networks that differ at most by a factor that is logarithmic in the network's width and linear in its depth. In the special case of shallow networks, we obtain matching bounds. Remarkably, the behavior of the $\ell^p$-Lipschitz constant varies significantly between the regimes $ p \in [1,2) $ and $ p \in [2,\infty] $. For $p \in [2,\infty]$, the $\ell^p$-Lipschitz constant behaves similarly to $\Vert g\Vert_{p'}$, where $g \in \mathbb{R}^d$ is a $d$-dimensional standard Gaussian vector and $1/p + 1/p' = 1$. In contrast, for $p \in [1,2)$, the $\ell^p$-Lipschitz constant aligns more closely to $\Vert g \Vert_{2}$.
Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Upper and lower bounds for the Lipschitz constant of random neural networksInformation and Inference: A Journal of the IMA, 2025
- Sharp Estimates on Random Hyperplane TessellationsSIAM Journal on Mathematics of Data Science, 2022
- Random version of Dvoretzky’s theorem inStochastic Processes and their Applications, 2017
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Measure Theory and Fine Properties of Functions, Revised Edition2015
- Dimension Reduction by Random Hyperplane TessellationsDiscrete & Computational Geometry, 2013