Abstract
The sample selection approach is very popular in learning with noisy labels. As deep networks “learn pattern first” , prior methods built on sample selection share a similar training procedure: the small-loss examples can be regarded as clean examples and used for helping generalization, while the large-loss examples are treated as mislabeled ones and excluded from network parameter updates. However, such a procedure is arguably debatable from two folds: (a) it does not consider the bad influence of noisy labels in selected small-loss examples; (b) it does not make good use of the discarded large-loss examples, which may be clean or have meaningful information for generalization. In this paper, we propose regularly truncated M-estimators (RTME) to address the above two issues simultaneously . Specifically, RTME can alternately switch modes between truncated M-estimators and original M-estimators . The former can adaptively select small-losses examples without knowing the noise rate and reduce the side-effects of noisy labels in them. The latter makes the possibly clean examples but with large losses involved to help generalization. Theoretically, we demonstrate that our strategies are label-noise-tolerant. Empirically, comprehensive experimental results show that our method can outperform multiple baselines and is robust to broad noise types and levels. The implementation is available at https://github.com/xiaoboxia/RTM_LNL .
Funding Information
  • National Natural Science Foundation of China (62336003, 12371510)
  • NSF of Jiangsu Province (BZ2021013, BK20220080)
  • NSFC Young Scientists Fund (62006202)
  • NSFC General Program (62376235)
  • RGC Early Career Scheme (22200720)
  • National Natural Science Foundation of China (62276242)
  • National Aviation Science Foundation (2022Z071078001)
  • CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-016B, CAAIXSJLJJ-2022-001A)
  • Anhui Province Key Research and Development Program (202104a05020007)
  • DreamsFoundation of Jianghuai Advance Technology Center (2023-ZM01Z001)
  • USTC-IAT Application Sci. & Tech. Achievement Cultivation Program (JL06521001Y)
  • Sci. & Tech. Innovation Special Zone (20-163-14-LZ-001-004-01)
  • National Natural Science Foundation of China (62125201, 62020106007)
  • Australian Research Council (FT220100318, DP220102121, LP220100527, LP220200949, IC190100031)