A Unifying Representer Theorem for Inverse Problems and Machine Learning
Author:
Open Access
- 25 September 2020
- journal article
- research article
- Published by Springer Nature in Foundations of Computational Mathematics
- Vol. 21 (4) , 941-960
Abstract
Regularization addresses the ill-posedness of the training problem in machine learning or the reconstruction of a signal from a limited number of measurements. The method is applicable whenever the problem is formulated as an optimization task. The standard strategy consists in augmenting the original cost functional by an energy that penalizes solutions with undesirable behavior. The effect of regularization is very well understood when the penalty involves a Hilbertian norm. Another popular configuration is the use of an $$\ell _1$$ -norm (or some variant thereof) that favors sparse solutions. In this paper, we propose a higher-level formulation of regularization within the context of Banach spaces. We present a general representer theorem that characterizes the solutions of a remarkably broad class of optimization problems. We then use our theorem to retrieve a number of known results in the literature such as the celebrated representer theorem of machine leaning for RKHS, Tikhonov regularization, representer theorems for sparsity promoting functionals, the recovery of spikes, as well as a few new ones.
Keywords
All Related Versions
Funding Information
- EPFL Lausanne
This publication has 47 references indexed in Scilit:
- Vector-valued reproducing kernel Banach spaces with applications to multi-task learningJournal of Complexity, 2013
- Semi-reproducing kernel Hilbert spaces, splines and increment krigingJournal of Nonparametric Statistics, 2010
- Kernels, regularization and differential equationsPattern Recognition, 2008
- On Learning Vector-Valued FunctionsNeural Computation, 2005
- The Generalized LASSOIEEE Transactions on Neural Networks, 2004
- Comparing support vector machines with Gaussian kernels to radial basis function classifiersIEEE Transactions on Signal Processing, 1997
- Regularization theory in image restoration-the stabilizing functional approachIEEE Transactions on Acoustics, Speech, and Signal Processing, 1990
- Regularization Algorithms for Learning That Are Equivalent to Multilayer NetworksScience, 1990
- Theory of Reproducing KernelsTransactions of the American Mathematical Society, 1950
- Sur les fonctions conjugu esMathematische Zeitschrift, 1928