On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation
Open Access
- 4 January 2017
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 35 (3) , 1-32
Abstract
Magnitude estimation is a psychophysical scaling technique for the measurement of sensation, where observers assign numbers to stimuli in response to their perceived intensity. We investigate the use of magnitude estimation for judging the relevance of documents for information retrieval evaluation, carrying out a large-scale user study across 18 TREC topics and collecting over 50,000 magnitude estimation judgments using crowdsourcing. Our analysis shows that magnitude estimation judgments can be reliably collected using crowdsourcing, are competitive in terms of assessor cost, and are, on average, rank-aligned with ordinal judgments made by expert relevance assessors. We explore the application of magnitude estimation for IR evaluation, calibrating two gain-based effectiveness metrics, nDCG and ERR, directly from user-reported perceptions of relevance. A comparison of TREC system effectiveness rankings based on binary, ordinal, and magnitude estimation relevance shows substantial variation; in particular, the top systems ranked using magnitude estimation and ordinal judgments differ substantially. Analysis of the magnitude estimation scores shows that this effect is due in part to varying perceptions of relevance: different users have different perceptions of the impact of relative differences in document relevance. These results have direct implications for IR evaluation, suggesting that current assumptions about a single view of relevance being sufficient to represent a population of users are unlikely to hold.Keywords
Funding Information
- Australian Research Council (DP130104007)
- Google Faculty Research Award
This publication has 34 references indexed in Scilit:
- How Much is too Much? Using Magnitude Estimation for User Experience ResearchProceedings of the Human Factors and Ergonomics Society Annual Meeting, 2010
- Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevanceJournal of the American Society for Information Science and Technology, 2007
- Answering the Call for a Standard Reliability Measure for Coding DataCommunication Methods and Measures, 2007
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- Towards the identification of the optimal number of relevance categoriesJournal of the American Society for Information Science, 1999
- Magnitude Estimation of Linguistic AcceptabilityLanguage, 1996
- Measuring relevance judgmentsInformation Processing & Management, 1988
- The Optimal Number of Response Alternatives for a Scale: A ReviewJournal of Marketing Research, 1980
- MAGNITUDE ESTIMATION: NOTES ON WHAT, HOW, WHEN, AND WHY TO USE ITJournal of Food Quality, 1977
- A Metric for the Social ConsensusScience, 1966