How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?
Preprint
Abstract
The capacity of deep learning models is often large enough to both learn the underlying statistical signal and overfit to noise in the training set. This noise memorization can be harmful especially for data with a low signal-to-noise ratio (SNR), leading to poor generalization. Inspired by prior observations that label noise provides implicit regularization that improves generalization, in this work, we investigate whether introducing label noise to the gradient updates can enhance the test performance of neural network (NN) in the low SNR regime. Specifically, we consider training a two-layer NN with a simple label noise gradient descent (GD) algorithm, in an idealized signal-noise data setting. We prove that adding label noise during training suppresses noise memorization, preventing it from dominating the learning process; consequently, label noise GD enjoys rapid signal growth while the overfitting remains controlled, thereby achieving good generalization despite the low SNR. In contrast, we also show that NN trained with standard GD tends to overfit to noise in the same low SNR setting and establish a non-vanishing lower bound on its test error, thus demonstrating the benefit of introducing label noise in gradient-based training.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- On the Comparison between Multi-modal and Single-modal Contrastive Learning2024
- Benign Overfitting and Noisy FeaturesJournal of the American Statistical Association, 2022
- High-Dimensional Limit Theorems for SGD: Effective Dynamics and Critical Scaling2022
- Understanding deep learning (still) requires rethinking generalizationCommunications of the ACM, 2021