Detecting noisy labels using checkpoint ensembles

Authors:
Open Access
Publisher website
Google Scholar
Abstract
We are concerned with the problem of identifying samples with noisy labels in a given dataset. Using the predictions of well-generalizing models to flag incorrectly predicted labels as noisy is a known method, but it is generally not considered competitive. At the same time, it has been observed recently that gradient descent fits clean samples first, and the noisy samples are memorized later. Inspired by related theoretical results, we revisit the idea of using the predictions of well-generalizing model checkpoints during training to classify samples as noisy or clean. We offer two key improvements that allow this strikingly simple approach to outperform some well-known methods. Our first method, ES-Prediction, classifies a sample as noisy if the model checkpoint predicts its label incorrectly. Our second method, ES-BGSS, is based on the observation that the sum of the per-sample gradients of the loss with respect to the feature layer bias parameters tends to be negative for clean samples and positive for noisy ones. Both methods are applied to an ensemble of model checkpoints collected around the early stopping point of a standard training run, which reduces the variance of the predictions and improves detection accuracy. Both methods operate in-sample, meaning they evaluate each checkpoint on its own training set and directly exploit the clean priority learning phenomenon. Our approaches only passively observe a normal training run and collect checkpoints. No extra input samples are added, and no pretrained models are used. Through empirical evaluations, we demonstrate that our methods are competitive with other approaches from related work for both detecting noisy samples and for noise-filtered training.
Funding Information
  • Nemzeti Kutatási, Fejlesztési és Innovaciós Alap (2024-1.2.3-HU-RIZONT-2024-00017)
  • Nemzeti Kutatási, Fejlesztési és Innovaciós Alap (RRF-2.3.1-21-2022-00004)

This publication has 21 references indexed in Scilit: