Continuous Review and Timely Correction: Enhancing the Resistance to Noisy Labels via Self-Not-True and Class-Wise Distillation
- 29 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (5) , 5165-5179
Abstract
Deep neural networks possess remarkable learning capabilities but are vulnerable to overfitting in the presence of mislabeled data. A well-known memorization effect causes networks to first fit clean samples and later memorize noisy labels. Although early stopping can partially alleviate this issue, it cannot prevent the accumulation of incorrect knowledge or recover information lost due to mislabeled inputs. In this paper, we introduce an innovative mechanism for continuous review and timely correction of learned knowledge. Our approach allows the network to repeatedly revisit and reinforce correct information while promptly addressing any inaccuracies stemming from mislabeled data. We present a novel method called self-not-true-distillation (SNTD). This technique employs self-distillation, where the network from previous training iterations acts as a teacher, guiding the current network to review and solidify its understanding of accurate labels. Crucially, SNTD masks the true class label in the logits during this process, concentrating on the non-true classes to correct any erroneous knowledge that may have been acquired. We also recognize that different data classes follow distinct learning trajectories. A single teacher network might struggle to effectively guide the learning of all classes at once, which necessitates selecting different teacher networks for each specific class. Additionally, the influence of the teacher network's guidance varies throughout the training process. To address these challenges, we propose SNTD+, which integrates a class-wise distillation strategy along with a dynamic weight adjustment mechanism. Together, these enhancements significantly bolster SNTD's robustness in tackling complex scenarios characterized by label noise.Keywords
Funding Information
- National Natural Science Foundation of China (62376282)
This publication has 31 references indexed in Scilit:
- Dynamics-aware loss for learning with label noisePattern Recognition, 2023
- ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks2021
- Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization2020
- Semi-online Multi-people Tracking by Re-identificationInternational Journal of Computer Vision, 2020
- Towards instance-dependent label noise-tolerant classification: a probabilistic approachPattern Analysis and Applications, 2018
- Joint Optimization Framework for Learning with Noisy Labels2018
- Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach2017
- Deep Residual Learning for Image Recognition2016
- Learning from massive noisy labeled data for image classification2015
- Noise resistant graph ranking for improved web image search2011