Empirical risk minimization in the interpolating regime with application to neural network learning
Open Access
- 21 February 2025
- journal article
- research article
- Published by Springer Nature in Machine Learning
- Vol. 114 (4) , 1-52
Abstract
A common strategy to train deep neural networks (DNNs) is to use very large architectures and to train them until they (almost) achieve zero training error. Empirically observed good generalization performance on test data, even in the presence of lots of label noise, corroborate such a procedure. On the other hand, in statistical learning theory it is known that over-fitting models may lead to poor generalization properties, occurring in e.g. empirical risk minimization (ERM) over too large hypotheses classes. Inspired by this contradictory behavior, so-called interpolation methods have recently received much attention, leading to consistent and optimally learning methods for, e.g., some local averaging schemes with zero training error. We extend this analysis to ERM-like methods for least squares regression and show that for certain, large hypotheses classes called inflated histograms, some interpolating empirical risk minimizers enjoy very good statistical guarantees while others fail in the worst sense. Moreover, we show that the same phenomenon occurs for DNNs with zero training error and sufficiently large architectures.Keywords
All Related Versions
Funding Information
- Technische Universität Braunschweig
This publication has 18 references indexed in Scilit:
- How many neurons do we need? A refined analysis for shallow networks trained with gradient descentJournal of Statistical Planning and Inference, 2024
- Estimation of a regression function on a manifold by fully connected deep neural networksJournal of Statistical Planning and Inference, 2022
- The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent CurveCommunications on Pure and Applied Mathematics, 2021
- Benign overfitting in linear regressionProceedings of the National Academy of Sciences, 2020
- Understanding Machine Learning2014
- Introduction to Nonparametric Estimation2009
- Adaptive regression estimation with multilayer feedforward neural networksJournal of Nonparametric Statistics, 2005
- A Distribution-Free Theory of Nonparametric Regression2002
- A Probabilistic Theory of Pattern Recognition1996
- Convergence rates for single hidden layer feedforward networksNeural Networks, 1994