Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms
Preprint
Abstract
We study theoretical properties of a broad class of regularized algorithms with vector-valued output. These spectral algorithms include kernel ridge regression, kernel principal component regression, various implementations of gradient descent and many more. Our contributions are twofold. First, we rigorously confirm the so-called saturation effect for ridge regression with vector-valued output by deriving a novel lower bound on learning rates; this bound is shown to be suboptimal when the smoothness of the regression function exceeds a certain level. Second, we present the upper bound for the finite sample risk general vector-valued spectral algorithms, applicable to both well-specified and misspecified scenarios (where the true regression function lies outside of the hypothesis space) which is minimax optimal in various regimes. All of our results explicitly allow the case of infinite-dimensional output variables, proving consistency of recent practical applications.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Optimal rates for spectral algorithms with least-squares regression over Hilbert spacesApplied and Computational Harmonic Analysis, 2020
- Optimal Rates for Regularization of Statistical Inverse Learning ProblemsFoundations of Computational Mathematics, 2017
- Multi-output learning via spectral filteringMachine Learning, 2012
- Mercer’s Theorem on General Domains: On the Interaction between Measures, Kernels, and RKHSsConstructive Approximation, 2012
- Global Saturation of Regularization Methods for Inverse Ill-Posed ProblemsJournal of Optimization Theory and Applications, 2010
- Learning Theory Estimates via Integral Operators and Their ApproximationsConstructive Approximation, 2007
- On regularization algorithms in learning theoryJournal of Complexity, 2007
- Optimal Rates for the Regularized Least-Squares AlgorithmFoundations of Computational Mathematics, 2006
- Shannon sampling II: Connections to learning theoryApplied and Computational Harmonic Analysis, 2005
- Reproducing Kernel Hilbert Spaces in Probability and Statistics2004