Multi-label Answer Aggregation Based on Joint Matrix Factorization
Authors:
- Jinzheng TuJTJinzheng Tu
- Guoxian YuGYGuoxian Yu
- Carlotta DomeniconiCDCarlotta Domeniconi
- Jun WangJWJun Wang
- Guoqiang XiaoGXGuoqiang Xiao
- Maozu GuoMGMaozu Guo
- 1 November 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 517-526
Abstract
Crowdsourcing is a useful and economic approach to data annotation. To obtain annotation of high quality, various aggregation approaches have been developed, which take into account different factors that impact the quality of aggregated answers. However, existing methods generally focus on single-label (multi-class and binary) tasks, and they ignore the inter-correlation between labels, and thus may have compromised quality. In this paper, we introduce a Multi-Label answer aggregation approach based on Joint Matrix Factorization (ML-JMF). ML-JMF selectively and jointly factorizes the sample-label association matrices collected from different annotators into products of individual and shared low-rank matrices. As such, it takes advantage of the robustness of low-rank matrix approximation to noise, and reduces the impact of unreliable annotators by assigning small (zero) weights to their annotation matrices. In addition, it takes advantage of the correlation among labels by leveraging the shared low-rank matrix, and of the similarity between annotators using the individual low-rank matrices to guide the factorization. ML-JMF pursues the low-rank matrices via a unified objective function, and introduces an iterative technique to optimize it. ML-JMF finally uses the optimized low-rank matrices and weights to infer the ground-truth labels. Our experimental results on multi-label datasets show that ML-JMF outperforms competitive methods in inferring ground truth labels. Our approach can identify unreliable annotators, and is robust against their misleading answers through the assignment of small (zero) weights to their annotation.Keywords
This publication has 31 references indexed in Scilit:
- Separate or joint? Estimation of multiple labels from crowdsourced annotationsExpert Systems with Applications, 2014
- Data Fusion by Matrix FactorizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Imbalanced Multiple Noisy LabelingIEEE Transactions on Knowledge and Data Engineering, 2014
- Pattern classification of dermoscopy images: A perceptually uniform modelPattern Recognition, 2013
- Matrix correlations for high-dimensional data: the modified RV-coefficientBioinformatics, 2008
- Fusion of Mass Spectrometry-Based Metabolomics DataAnalytical Chemistry, 2005
- Discriminative Methods for Multi-labeled Classification2004
- Noise estimation and filtering using block-based singular value decompositionIEEE Transactions on Image Processing, 1997
- An argument for basic emotionsCognition and Emotion, 1992
- Maximum Likelihood Estimation of Observer Error-Rates Using the EM AlgorithmJournal of the Royal Statistical Society Series C: Applied Statistics, 1979