What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory
Preprint
Abstract
We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit activations to data. We propose a new function space, which is reminiscent of classical bounded variation-type spaces, that captures the compositional structure associated with deep neural networks. We derive a representer theorem showing that deep ReLU networks are solutions to regularized data fitting problems over functions from this space. The function space consists of compositions of functions from the Banach spaces of second-order bounded variation in the Radon domain. These are Banach spaces with sparsity-promoting norms, giving insight into the role of sparsity in deep neural networks. The neural network solutions have skip connections and rank bounded weight matrices, providing new theoretical support for these common architectural choices. The variational problem we study can be recast as a finite-dimensional neural network training problem with regularization schemes related to the notions of weight decay and path-norm regularization. Finally, our analysis builds on techniques from variational spline theory, providing new connections between deep neural networks and splines.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Convex optimization in sums of Banach spacesApplied and Computational Harmonic Analysis, 2022
- Mad Max: Affine Spline Insights Into Deep LearningProceedings of the IEEE, 2020
- Learning Activation Functions in Deep (Spline) Neural NetworksIEEE Open Journal of Signal Processing, 2020
- The Role of Neural Network Activation FunctionsIEEE Signal Processing Letters, 2020
- A Unifying Representer Theorem for Inverse Problems and Machine LearningFoundations of Computational Mathematics, 2020
- Deep Convolutional Neural Network for Inverse Problems in ImagingIEEE Transactions on Image Processing, 2017
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Vector-valued Lg-splines I. Interpolating splinesJournal of Mathematical Analysis and Applications, 1979
- Spline solutions to L1 extremal problems in one and several variablesJournal of Approximation Theory, 1975
- Some results on Tchebycheffian spline functionsJournal of Mathematical Analysis and Applications, 1971