Robust Subjective Visual Property Prediction from Crowdsourced Pairwise Labels
Authors:
- Yanwei FuYFYanwei Fu
- Timothy M. HospedalesTHTimothy M. Hospedales
- Tao XiangTXTao Xiang
- Jiechao XiongJXJiechao Xiong
- Shaogang GongSGShaogang Gong
- Yizhou WangYWYizhou Wang
- Yuan YaoYYYuan Yao
Preprint
Abstract
The problem of estimating subjective visual properties from image and video has attracted increasing interest. A subjective visual property is useful either on its own (e.g. image and video interestingness) or as an intermediate representation for visual recognition (e.g. a relative attribute). Due to its ambiguous nature, annotating the value of a subjective visual property for learning a prediction model is challenging. To make the annotation more reliable, recent studies employ crowdsourcing tools to collect pairwise comparison labels because human annotators are much better at ranking two images/videos (e.g. which one is more interesting) than giving an absolute value to each of them separately. However, using crowdsourced data also introduces outliers. Existing methods rely on majority voting to prune the annotation outliers/errors. They thus require large amount of pairwise labels to be collected. More importantly as a local outlier detection method, majority voting is ineffective in identifying outliers that can cause global ranking inconsistencies. In this paper, we propose a more principled way to identify annotation outliers by formulating the subjective visual property prediction task as a unified robust learning to rank problem, tackling both the outlier detection and learning to rank jointly. Differing from existing methods, the proposed method integrates local pairwise comparison labels together to minimise a cost that corresponds to global inconsistency of ranking order. This not only leads to better detection of annotation outliers but also enables learning with extremely sparse annotations. Extensive experiments on various benchmark datasets demonstrate that our new approach significantly outperforms state-of-the-arts alternatives.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Interestingness Prediction by Robust Learning to Rank2014
- Robust relative attributes for human action recognitionPattern Analysis and Applications, 2013
- Scaled sparse linear regressionBiometrika, 2012
- Attributes for Classifier Feedback2012
- Efficient Bayesian Inference for Generalized Bradley–Terry ModelsJournal of Computational and Graphical Statistics, 2012
- Square-root lasso: pivotal recovery of sparse signals via conic programmingBiometrika, 2011
- Statistical ranking and combinatorial Hodge theoryMathematical Programming, 2010
- Efficient algorithms for ranking with SVMsInformation Retrieval Journal, 2009
- Robust estimation and wavelet thresholding in partially linear modelsStatistics and Computing, 2007
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial EnvelopeInternational Journal of Computer Vision, 2001