Active Learning With Imbalanced Multiple Noisy Labeling
Authors:
- 14 August 2014
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 45 (5) , 1095-1107
Abstract
With crowdsourcing systems, it is easy to collect multiple noisy labels for the same object for supervised learning. This dynamic annotation procedure fits the active learning perspective and accompanies the imbalanced multiple noisy labeling problem. This paper proposes a novel active learning framework with multiple imperfect annotators involved in crowdsourcing systems. The framework contains two core procedures: label integration and instance selection. In the label integration procedure, a positive label threshold (PLAT) algorithm is introduced to induce the class membership from the multiple noisy label set of each instance in a training set. PLAT solves the imbalanced labeling problem by dynamically adjusting the threshold for determining the class membership of an example. Furthermore, three novel instance selection strategies are proposed to adapt PLAT for improving the learning performance. These strategies are respectively based on the uncertainty derived from the multiple labels, the uncertainty derived from the learned model, and the combination method (CFI). Experimental results on 12 datasets with different underlying class distributions demonstrate that the three novel instance selection strategies significantly improve the learning performance, and CFI has the best performance when labeling behaviors exhibit different levels of imbalance in crowdsourcing systems. We also apply our methods to a real-world scenario, obtaining noisy labels from Amazon Mechanical Turk, and show that our proposed strategies achieve very high performance.Keywords
Funding Information
- Program for Changjiang Scholars and Innovative Research Team in University (PCSIRT) of the Ministry of Education, China (IRT13059)
- National 973 Program of China (2013CB329604)
- National 863 Program of China (2012AA011005)
- National Natural Science Foundation of China (61229301)
- U.S. National Science Foundation (IIS-1115417)
This publication has 18 references indexed in Scilit:
- A threshold method for imbalanced multiple noisy labeling2013
- Repeated labeling using multiple noisy labelersData Mining and Knowledge Discovery, 2013
- On measuring the performance of binary classifiersKnowledge and Information Systems, 2012
- Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations2011
- Active Learning From Stream Data Using Optimal Weight Classifier EnsembleIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2010
- Learning from Imbalanced DataIEEE Transactions on Knowledge and Data Engineering, 2009
- Proactive learning2008
- Active learning with direct query construction2008
- Large-scale text categorization by batch mode active learning2006
- Active Hidden Markov Models for Information Extraction2001