Fixed points of arbitrarily deep 1-dimensional neural networks
Preprint
Abstract
In this paper, we establish a sharp upper bound on the the number of fixed points a certain class of neural networks can have. The networks under study (autoencoders) can be viewed as discrete dynamical systems whose nonlinearities are given by the choice of activation functions. To this end, we introduce a new class $\mathcal{F}$ of $C^1$ activation functions that is closed under composition, and contains e.g. the logistic sigmoid function. We use this class to show that any 1-dimensional neural network of arbitrary depth with activation functions in $\mathcal{F}$ has at most three fixed points. Due to the simple nature of such networks, we are able to completely understand their fixed points, providing a foundation to the much needed connection between application and theory of deep neural networks.
Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Autoencoder Attractors for Uncertainty Estimation2022
- Feedback Recurrent Autoencoder2020
- Ordinary Differential Equations2002
- Stable Orbits and Bifurcation of Maps of the IntervalSIAM Journal on Applied Mathematics, 1978
- Half-order differentials on Riemann surfacesActa Mathematica, 1966