On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks
Preprint
Abstract
Let F:R^n -> R be a feedforward ReLU neural network. It is well-known that for any choice of parameters, F is continuous and piecewise (affine) linear. We lay some foundations for a systematic investigation of how the architecture of F impacts the geometry and topology of its possible decision regions for binary classification tasks. Following the classical progression for smooth functions in differential topology, we first define the notion of a generic, transversal ReLU neural network and show that almost all ReLU networks are generic and transversal. We then define a partially-oriented linear 1-complex in the domain of F and identify properties of this complex that yield an obstruction to the existence of bounded connected components of a decision region. We use this obstruction to prove that a decision region of a generic, transversal ReLU network F: R^n -> R with a single hidden layer of dimension (n + 1) can have no more than one bounded connected component.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- On decision regions of narrow deep neural networksNeural Networks, 2021
- On the Complexity of Neural Network Classifiers: A Comparison Between Shallow and Deep ArchitecturesIEEE Transactions on Neural Networks and Learning Systems, 2014
- Approximation capabilities of multilayer feedforward networksNeural Networks, 1991
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989
- Facing up to arrangements: face-count formulas for partitions of space by hyperplanesMemoirs of the American Mathematical Society, 1975