Trustable Co-Label Learning From Multiple Noisy Annotators
- 23 December 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 1045-1057
Abstract
Supervised deep learning depends on massive accurately annotated examples, which is usually impractical in many real-world scenarios. A typical alternative is learning from multiple noisy annotators. Numerous earlier works assume that all labels are noisy, while it is usually the case that a few trusted samples with clean labels are available. This raises the following important question: how can we effectively use a small amount of trusted data to facilitate robust classifier learning from multiple annotators This paper proposes a data-efficient approach, called Trustable Co-label Learning (TCL), to learn deep classifiers from multiple noisy annotators when a small set of trusted data is available. This approach follows the coupled-view learning manner, which jointly learns the data classifier and the label aggregator. It effectively uses trusted data as a guide to generate trustable soft labels (termed co-labels). A co-label learning can then be performed by alternately reannotating the pseudo labels and refining the classifiers. In addition, we further improve TCL for a special complete data case, where each instance is labeled by all annotators and the label aggregator is represented by multilayer neural networks to enhance model capacity. Extensive experiments on synthetic and real datasets clearly demonstrate the effectiveness and robustness of the proposed approach. The code will be available at https://github.com/ShikunLi/TCL.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61772513)
- Beijing Natural Science Foundation (L192040)
- National Key Research and Development Plan (2020AAA0140001)
- Youth Innovation Promotion Association
- Chinese Academy of Sciences
This publication has 64 references indexed in Scilit:
- MODEL AVERAGING IN ECONOMICS: AN OVERVIEWJournal of Economic Surveys, 2013
- A weight-adjusted voting algorithm for ensembles of classifiersJournal of the Korean Statistical Society, 2011
- Frequentist model averaging estimation: a reviewJournal of Systems Science and Complexity, 2009
- Locally adaptive metrics for clustering high dimensional dataData Mining and Knowledge Discovery, 2007
- Decision templates for multiple classifier fusion: an experimental comparisonPattern Recognition, 2001
- Combining multiple neural networks by fuzzy integral for robust classificationIEEE Transactions on Systems, Man, and Cybernetics, 1995
- Stacked generalizationNeural Networks, 1992
- A review of fuzzy set aggregation connectivesInformation Sciences, 1985
- Comparison of Discrimination Techniques Applied to a Complex Data Set of Head Injured PatientsJournal of the Royal Statistical Society. Series A (General), 1981
- Maximum Likelihood Estimation of Observer Error-Rates Using the EM AlgorithmJournal of the Royal Statistical Society Series C: Applied Statistics, 1979