What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory
- 13 April 2022
- journal article
- research article
- Published by Society for Industrial & Applied Mathematics (SIAM) in SIAM Journal on Mathematics of Data Science
- Vol. 4 (2) , 464-489
Abstract
We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit (ReLU) activations to data. We propose a new function...Keywords
All Related Versions
Funding Information
- Air Force Office of Scientific Research (FA9550-18-1-0166)
- Office of Naval Research (N00014-20-1-2787)
- National Science Foundation (DGE-1747503)
This publication has 17 references indexed in Scilit:
- A Unifying Representer Theorem for Inverse Problems and Machine LearningFoundations of Computational Mathematics, 2020
- Deep Neural Networks With Trainable Activations and Controlled Lipschitz ConstantIEEE Transactions on Signal Processing, 2020
- Deep Convolutional Neural Network for Inverse Problems in ImagingIEEE Transactions on Image Processing, 2017
- Splines Are Universal Solutions of Linear Inverse Problems with Generalized TV RegularizationSIAM Review, 2017
- Deep learningNature, 2015
- Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research GroupsIEEE Signal Processing Magazine, 2012
- Vector-valued Lg-splines I. Interpolating splinesJournal of Mathematical Analysis and Applications, 1979
- Spline solutions to L1 extremal problems in one and several variablesJournal of Approximation Theory, 1975
- Some results on Tchebycheffian spline functionsJournal of Mathematical Analysis and Applications, 1971
- Theory of reproducing kernelsTransactions of the American Mathematical Society, 1950