Generalisation error in learning with random features and the hidden manifold model*
Authors:
- 1 December 2021
- journal article
- research article
- Published by IOP Publishing in Journal of Statistical Mechanics: Theory and Experiment
- Vol. 2021 (12) , 124013
Abstract
We study generalised linear regression and classification for a synthetically generated dataset encompassing different problems of interest, such as learning with random features, neural networks in the lazy training regime, and the hidden manifold model. We consider the high-dimensional regime and using the replica method from statistical physics, we provide a closed-form expression for the asymptotic generalisation performance in these problems, valid in both the under- and over-parametrised regimes and for a broad choice of generalised linear model loss functions. In particular, we show how to obtain analytically the so-called double descent behaviour for logistic regression with a peak at the interpolation threshold, we illustrate the superiority of orthogonal against random Gaussian projections in learning with random features, and discuss the role played by correlations in the data generated by the hidden manifold model. Beyond the interest in these particular problems, the theoretical formalism introduced in this manuscript provides a path to further extensions to more complex tasks.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Statistical mechanics of complex neural systems and high dimensional dataJournal of Statistical Mechanics: Theory and Experiment, 2013
- Statistical-Physics-Based Reconstruction in Compressed SensingPhysical Review X, 2012
- A typical reconstruction limit for compressed sensing based on Lp-norm minimizationJournal of Statistical Mechanics: Theory and Experiment, 2009
- Deterministic equivalents for certain functionals of large random matricesThe Annals of Applied Probability, 2007
- Optimal Rates for the Regularized Least-Squares AlgorithmFoundations of Computational Mathematics, 2006
- The Parisi formulaAnnals of Mathematics, 2006
- The statistical mechanics of learning a ruleReviews of Modern Physics, 1993
- Statistical mechanics of learning from examplesPhysical Review A, 1992
- Neural Networks and the Bias/Variance DilemmaNeural Computation, 1992
- Three unfinished works on the optimal storage capacity of networksJournal of Physics A: General Physics, 1989