Ingo Steinwart
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Ingo Steinwart has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
Clint Scovel
5 shared publicationsDon Hush
4 shared publications- Andreas Christmann
University of Bayreuth
3 shared publications - Clint Scovel
Los Alamos National Laboratory
3 shared publications - David Holzmüller
University of Stuttgart
3 shared publications - Johan A. K. Suykens
ESAT-STADIUS, KU Leuven
3 shared publications
Latest Publications
- Open Access
- 15 October 2026
- Published by Springer Nature in Constructive Approximation
- p. 1-39
Spectral Representations of Interpolation Spaces of Reproducing Kernel Hilbert Spaces
In statistical learning theory, interpolation spaces of the form $$[\textrm{L}^2,H]_{\theta ,r}$$ , where H is a reproducing kernel Hilbert space, are in widespread use. So far, however, they are only well understood for fine index $$r=2$$ . We generalise existing results from $$r=2$$ to $$1\le r \le \infty $$ . In particular, we present a spectral decomposition of such spaces, analyse their embedding properties, and describe connections to the theory of Banach spaces of functions. Additionally, we present example applications of our results to regularisation error estimation in statistical learning.- 15 October 2026
- Published by Springer Nature in Constructive Approximation
- p. 1-39
Open AccessSpectral Representations of Interpolation Spaces of Reproducing Kernel Hilbert Spaces
In statistical learning theory, interpolation spaces of the form $$[\textrm{L}^2,H]_{\theta ,r}$$ , where H is a reproducing kernel Hilbert space, are in widespread use. So far, however, they are only well understood for fine index $$r=2$$ . We generalise existing results from $$r=2$$ to $$1\le r \le \infty $$ . In particular, we present a spectral decomposition of such spaces, analyse their embedding properties, and describe connections to the theory of Banach spaces of functions. Additionally, we present example applications of our results to regularisation error estimation in statistical learning. - 1 October 2026
- Published by Institute of Mathematical Statistics in The Annals of Applied Probability
- Vol. 36 (5) , 3995-4040
Conditioning of Banach space valued Gaussian random variables: An approximation approach based on martingales
We investigate the conditional distributions of two Banach space valued, jointly Gaussian random variables. In particular, we show that these conditional distributions are again Gaussian and that their means and covariances can be determined by a general finite-dimensional approximation scheme. Here, it turns out that the covariance operators occurring in this scheme converge with respect to the nuclear norm and that the conditional probabilities converge weakly. Furthermore, we discuss how our approximation scheme can be implemented in several classes of important Banach spaces such as (reproducing kernel) Hilbert spaces, spaces of continuous functions, and other spaces consisting of functions. As an example, we then apply our general results to the case of continuous Gaussian processes that are conditioned to partial but infinite observations of their paths. Here we show that conditioning on sufficiently rich, increasing sets of finitely many observations leads to consistent approximations, that is, both the mean and covariance functions converge uniformly and the conditional probabilities converge weakly. Moreover, we discuss how these results improve our understanding of the popular Gaussian processes for machine learning. From a technical perspective our results are based upon a Banach space valued martingale approach for regular conditional probabilities.
- 1 October 2026
- Published by Institute of Mathematical Statistics in The Annals of Applied Probability
- Vol. 36 (5) , 3995-4040
Conditioning of Banach space valued Gaussian random variables: An approximation approach based on martingales
We investigate the conditional distributions of two Banach space valued, jointly Gaussian random variables. In particular, we show that these conditional distributions are again Gaussian and that their means and covariances can be determined by a general finite-dimensional approximation scheme. Here, it turns out that the covariance operators occurring in this scheme converge with respect to the nuclear norm and that the conditional probabilities converge weakly. Furthermore, we discuss how our approximation scheme can be implemented in several classes of important Banach spaces such as (reproducing kernel) Hilbert spaces, spaces of continuous functions, and other spaces consisting of functions. As an example, we then apply our general results to the case of continuous Gaussian processes that are conditioned to partial but infinite observations of their paths. Here we show that conditioning on sufficiently rich, increasing sets of finitely many observations leads to consistent approximations, that is, both the mean and covariance functions converge uniformly and the conditional probabilities converge weakly. Moreover, we discuss how these results improve our understanding of the popular Gaussian processes for machine learning. From a technical perspective our results are based upon a Banach space valued martingale approach for regular conditional probabilities.
- Open Access
- 28 September 2026
- Published by Springer Nature in Constructive Approximation
- p. 1-49
Convergence Analysis of a Greedy Algorithm for Conditioning Gaussian Random Variables
In the context of Gaussian conditioning, greedy algorithms iteratively select the most informative measurements, given an observed Gaussian random variable. However, the convergence analysis for conditioning Gaussian random variables remains an open problem. We address this by introducing an operator M that allows us to transfer convergence rates of the observed Gaussian random variable approximation onto the conditional Gaussian random variable. Furthermore, we apply greedy methods from approximation theory to obtain convergence rates. These greedy methods have already demonstrated optimal convergence rates within the setting of kernel-based function approximation. In this paper, we establish an upper bound on the convergence rates concerning the norm of the approximation error of the conditional covariance operator.
- 28 September 2026
- Published by Springer Nature in Constructive Approximation
- p. 1-49
Open AccessConvergence Analysis of a Greedy Algorithm for Conditioning Gaussian Random Variables
In the context of Gaussian conditioning, greedy algorithms iteratively select the most informative measurements, given an observed Gaussian random variable. However, the convergence analysis for conditioning Gaussian random variables remains an open problem. We address this by introducing an operator M that allows us to transfer convergence rates of the observed Gaussian random variable approximation onto the conditional Gaussian random variable. Furthermore, we apply greedy methods from approximation theory to obtain convergence rates. These greedy methods have already demonstrated optimal convergence rates within the setting of kernel-based function approximation. In this paper, we establish an upper bound on the convergence rates concerning the norm of the approximation error of the conditional covariance operator.
- Open Access
- 21 February 2025
- Published by Springer Nature in Machine Learning
- Vol. 114 (4) , 1-52
Empirical risk minimization in the interpolating regime with application to neural network learning
A common strategy to train deep neural networks (DNNs) is to use very large architectures and to train them until they (almost) achieve zero training error. Empirically observed good generalization performance on test data, even in the presence of lots of label noise, corroborate such a procedure. On the other hand, in statistical learning theory it is known that over-fitting models may lead to poor generalization properties, occurring in e.g. empirical risk minimization (ERM) over too large hypotheses classes. Inspired by this contradictory behavior, so-called interpolation methods have recently received much attention, leading to consistent and optimally learning methods for, e.g., some local averaging schemes with zero training error. We extend this analysis to ERM-like methods for least squares regression and show that for certain, large hypotheses classes called inflated histograms, some interpolating empirical risk minimizers enjoy very good statistical guarantees while others fail in the worst sense. Moreover, we show that the same phenomenon occurs for DNNs with zero training error and sufficiently large architectures.
- 21 February 2025
- Published by Springer Nature in Machine Learning
- Vol. 114 (4) , 1-52
Open AccessEmpirical risk minimization in the interpolating regime with application to neural network learning
A common strategy to train deep neural networks (DNNs) is to use very large architectures and to train them until they (almost) achieve zero training error. Empirically observed good generalization performance on test data, even in the presence of lots of label noise, corroborate such a procedure. On the other hand, in statistical learning theory it is known that over-fitting models may lead to poor generalization properties, occurring in e.g. empirical risk minimization (ERM) over too large hypotheses classes. Inspired by this contradictory behavior, so-called interpolation methods have recently received much attention, leading to consistent and optimally learning methods for, e.g., some local averaging schemes with zero training error. We extend this analysis to ERM-like methods for least squares regression and show that for certain, large hypotheses classes called inflated histograms, some interpolating empirical risk minimizers enjoy very good statistical guarantees while others fail in the worst sense. Moreover, we show that the same phenomenon occurs for DNNs with zero training error and sufficiently large architectures.
- Open Access
Convergence Analysis of a Greedy Algorithm for Conditioning Gaussian Random Variables
In the context of Gaussian conditioning, greedy algorithms iteratively select the most informative measurements, given an observed Gaussian random variable. However, the convergence analysis for conditioning Gaussian random variables remains an open problem. We adress this by introducing an operator $M$ that allows us to transfer convergence rates of the observed Gaussian random variable approximation onto the conditional Gaussian random variable. Furthermore we apply greedy methods from approximation theory to obtain convergence rates. These greedy methods have already demonstrated optimal convergence rates within the setting of kernel based function approximation. In this paper, we establish an upper bound on the convergence rates concerning the norm of the approximation error of the conditional covariance operator.Open AccessConvergence Analysis of a Greedy Algorithm for Conditioning Gaussian Random Variables
In the context of Gaussian conditioning, greedy algorithms iteratively select the most informative measurements, given an observed Gaussian random variable. However, the convergence analysis for conditioning Gaussian random variables remains an open problem. We adress this by introducing an operator $M$ that allows us to transfer convergence rates of the observed Gaussian random variable approximation onto the conditional Gaussian random variable. Furthermore we apply greedy methods from approximation theory to obtain convergence rates. These greedy methods have already demonstrated optimal convergence rates within the setting of kernel based function approximation. In this paper, we establish an upper bound on the convergence rates concerning the norm of the approximation error of the conditional covariance operator. - Open Access
- 24 January 2025
- Published by Springer Nature in Constructive Approximation
- Vol. 63 (1) , 105-152
Which Spaces can be Embedded in Reproducing Kernel Hilbert Spaces?
Given a Banach space E consisting of functions, we ask whether there exists a reproducing kernel Hilbert space H with bounded kernel such that $$E\subset H$$ . More generally, we consider the question, whether for a given Banach space consisting of functions F with $$E\subset F$$ , there exists an intermediate reproducing kernel Hilbert space $$E\subset H\subset F$$ . We provide both sufficient and necessary conditions for this to hold. Moreover, we show that for typical classes of function spaces described by smoothness there is a strong dependence on the underlying dimension: the smoothness s required for the space E needs to grow proportional to the dimension d in order to allow for an intermediate reproducing kernel Hilbert space H .- 24 January 2025
- Published by Springer Nature in Constructive Approximation
- Vol. 63 (1) , 105-152
Open AccessWhich Spaces can be Embedded in Reproducing Kernel Hilbert Spaces?
Given a Banach space E consisting of functions, we ask whether there exists a reproducing kernel Hilbert space H with bounded kernel such that $$E\subset H$$ . More generally, we consider the question, whether for a given Banach space consisting of functions F with $$E\subset F$$ , there exists an intermediate reproducing kernel Hilbert space $$E\subset H\subset F$$ . We provide both sufficient and necessary conditions for this to hold. Moreover, we show that for typical classes of function spaces described by smoothness there is a strong dependence on the underlying dimension: the smoothness s required for the space E needs to grow proportional to the dimension d in order to allow for an intermediate reproducing kernel Hilbert space H . - Open Access
Physics-Informed Gaussian Process Regression Generalizes Linear PDE Solvers
Linear partial differential equations (PDEs) are an important, widely applied class of mechanistic models, describing physical processes such as heat transfer, electromagnetism, and wave propagation. In practice, specialized numerical methods based on discretization are used to solve PDEs. They generally use an estimate of the unknown model parameters and, if available, physical measurements for initialization. Such solvers are often embedded into larger scientific models with a downstream application and thus error quantification plays a key role. However, by ignoring parameter and measurement uncertainty, classical PDE solvers may fail to produce consistent estimates of their inherent approximation error. In this work, we approach this problem in a principled fashion by interpreting solving linear PDEs as physics-informed Gaussian process (GP) regression. Our framework is based on a key generalization of the Gaussian process inference theorem to observations made via an arbitrary bounded linear operator. Crucially, this probabilistic viewpoint allows to (1) quantify the inherent discretization error; (2) propagate uncertainty about the model parameters to the solution; and (3) condition on noisy measurements. Demonstrating the strength of this formulation, we prove that it strictly generalizes methods of weighted residuals, a central class of PDE solvers including collocation, finite volume, pseudospectral, and (generalized) Galerkin methods such as finite element and spectral methods. This class can thus be directly equipped with a structured error estimate. In summary, our results enable the seamless integration of mechanistic models as modular building blocks into probabilistic models by blurring the boundaries between numerical analysis and Bayesian inference.
Open AccessPhysics-Informed Gaussian Process Regression Generalizes Linear PDE Solvers
Linear partial differential equations (PDEs) are an important, widely applied class of mechanistic models, describing physical processes such as heat transfer, electromagnetism, and wave propagation. In practice, specialized numerical methods based on discretization are used to solve PDEs. They generally use an estimate of the unknown model parameters and, if available, physical measurements for initialization. Such solvers are often embedded into larger scientific models with a downstream application and thus error quantification plays a key role. However, by ignoring parameter and measurement uncertainty, classical PDE solvers may fail to produce consistent estimates of their inherent approximation error. In this work, we approach this problem in a principled fashion by interpreting solving linear PDEs as physics-informed Gaussian process (GP) regression. Our framework is based on a key generalization of the Gaussian process inference theorem to observations made via an arbitrary bounded linear operator. Crucially, this probabilistic viewpoint allows to (1) quantify the inherent discretization error; (2) propagate uncertainty about the model parameters to the solution; and (3) condition on noisy measurements. Demonstrating the strength of this formulation, we prove that it strictly generalizes methods of weighted residuals, a central class of PDE solvers including collocation, finite volume, pseudospectral, and (generalized) Galerkin methods such as finite element and spectral methods. This class can thus be directly equipped with a structured error estimate. In summary, our results enable the seamless integration of mechanistic models as modular building blocks into probabilistic models by blurring the boundaries between numerical analysis and Bayesian inference.
- Open Access
- 23 March 2024
- Published by Springer Nature in Archiv der Mathematik
- Vol. 122 (5) , 553-557
Reproducing kernel Hilbert spaces cannot contain all continuous functions on a compact metric space
Given an uncountable, compact metric space X, we show that there exists no reproducing kernel Hilbert space that contains the space of all continuous functions on X.
- 23 March 2024
- Published by Springer Nature in Archiv der Mathematik
- Vol. 122 (5) , 553-557
Open AccessReproducing kernel Hilbert spaces cannot contain all continuous functions on a compact metric space
Given an uncountable, compact metric space X, we show that there exists no reproducing kernel Hilbert space that contains the space of all continuous functions on X.
- Open Access
Which Spaces can be Embedded in Reproducing Kernel Hilbert Spaces?
Given a Banach space $E$ consisting of functions, we ask whether there exists a reproducing kernel Hilbert space $H$ with bounded kernel such that $E\subset H$. More generally, we consider the question, whether for a given Banach space consisting of functions $F$ with $E\subset F$, there exists an intermediate reproducing kernel Hilbert space $E\subset H\subset F$. We provide both sufficient and necessary conditions for this to hold. Moreover, we show that for typical classes of function spaces described by smoothness there is a strong dependence on the underlying dimension: the smoothness $s$ required for the space $E$ needs to grow \emph{proportional} to the dimension $d$ in order to allow for an intermediate reproducing kernel Hilbert space $H$.Open AccessWhich Spaces can be Embedded in Reproducing Kernel Hilbert Spaces?
Given a Banach space $E$ consisting of functions, we ask whether there exists a reproducing kernel Hilbert space $H$ with bounded kernel such that $E\subset H$. More generally, we consider the question, whether for a given Banach space consisting of functions $F$ with $E\subset F$, there exists an intermediate reproducing kernel Hilbert space $E\subset H\subset F$. We provide both sufficient and necessary conditions for this to hold. Moreover, we show that for typical classes of function spaces described by smoothness there is a strong dependence on the underlying dimension: the smoothness $s$ required for the space $E$ needs to grow \emph{proportional} to the dimension $d$ in order to allow for an intermediate reproducing kernel Hilbert space $H$. - Open Access
A Framework and Benchmark for Deep Batch Active Learning for Regression
The acquisition of labels for supervised learning can be expensive. To improve the sample efficiency of neural network regression, we study active learning methods that adaptively select batches of unlabeled data for labeling. We present a framework for constructing such methods out of (network-dependent) base kernels, kernel transformations, and selection methods. Our framework encompasses many existing Bayesian methods based on Gaussian process approximations of neural networks as well as non-Bayesian methods. Additionally, we propose to replace the commonly used last-layer features with sketched finite-width neural tangent kernels and to combine them with a novel clustering method. To evaluate different methods, we introduce an open-source benchmark consisting of 15 large tabular regression data sets. Our proposed method outperforms the state-of-the-art on our benchmark, scales to large data sets, and works out-of-the-box without adjusting the network architecture or training code. We provide open-source code that includes efficient implementations of all kernels, kernel transformations, and selection methods, and can be used for reproducing our results.
Open AccessA Framework and Benchmark for Deep Batch Active Learning for Regression
The acquisition of labels for supervised learning can be expensive. To improve the sample efficiency of neural network regression, we study active learning methods that adaptively select batches of unlabeled data for labeling. We present a framework for constructing such methods out of (network-dependent) base kernels, kernel transformations, and selection methods. Our framework encompasses many existing Bayesian methods based on Gaussian process approximations of neural networks as well as non-Bayesian methods. Additionally, we propose to replace the commonly used last-layer features with sketched finite-width neural tangent kernels and to combine them with a novel clustering method. To evaluate different methods, we introduce an open-source benchmark consisting of 15 large tabular regression data sets. Our proposed method outperforms the state-of-the-art on our benchmark, scales to large data sets, and works out-of-the-box without adjusting the network architecture or training code. We provide open-source code that includes efficient implementations of all kernels, kernel transformations, and selection methods, and can be used for reproducing our results.