Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed
Preprint
Abstract
Gaussian process regression underpins countless academic and industrial applications of machine learning and statistics, with maximum likelihood estimation routinely used to select appropriate parameters for the covariance kernel. However, it remains an open problem to establish the circumstances in which maximum likelihood estimation is well-posed, that is, when the predictions of the regression model are insensitive to small perturbations of the data. This article identifies scenarios where the maximum likelihood estimator fails to be well-posed, in that the predictive distributions are not Lipschitz in the data with respect to the Hellinger distance. These failure cases occur in the noiseless data setting, for any Gaussian process with a stationary covariance function whose lengthscale parameter is estimated using maximum likelihood. Although the failure of maximum likelihood estimation is part of Gaussian process folklore, these rigorous theoretical results appear to be the first of their kind. The implication of these negative results is that well-posedness may need to be assessed post-hoc, on a case-by-case basis, when maximum likelihood estimation is used to train a Gaussian process model.Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- Adaptive Radial Basis Function Partition of Unity Interpolation: A Bivariate Algorithm for Unstructured DataJournal of Scientific Computing, 2021
- Convergence Types and Rates in Generic Karhunen-Loève Expansions with Applications to Sample Path PropertiesPotential Analysis, 2018
- Multivariate interpolation with increasingly flat radial basis functions of finite smoothnessAdvances in Computational Mathematics, 2011
- An alternative procedure for selecting a good value for the parameter c in RBF-interpolationAdvances in Computational Mathematics, 2010
- Some Properties of Gaussian Reproducing Kernel Hilbert Spaces and Their Implications for Function Approximation and Learning TheoryConstructive Approximation, 2009
- Reproducing Kernel Hilbert Spaces Associated with Analytic Translation-Invariant Mercer KernelsJournal of Fourier Analysis and Applications, 2008
- Linearly constrained reconstruction of functions by kernels with applications to machine learningAdvances in Computational Mathematics, 2006
- Approximate Interpolation with Applications to Selecting Smoothing ParametersNumerische Mathematik, 2005
- Reproducing Kernel Hilbert Spaces in Probability and Statistics2004
- Error estimates and condition numbers for radial basis function interpolationAdvances in Computational Mathematics, 1995