BargCrEx: A System for Bargaining Based Aggregation of Crowd and Expert Opinions in Crowdsourcing
- 21 May 2022
- journal article
- research article
- Published by Springer Nature in Group Decision and Negotiation
- Vol. 31 (4) , 789-818
Abstract
Crowdsourcing and crowd voting systems are being increasingly used in societal, industry, and academic problems (labeling, recommendations, social choice, etc.) due to their possibility to exploit “wisdom of crowd” and obtain good quality solutions, and/or voter satisfaction, with high cost-efficiency. However, the decisions based on crowd vote aggregation do not guarantee high-quality results due to crowd voter data quality. Additionally, such decisions often do not satisfy the majority of voters due to data heterogeneity (multimodal or uniform vote distributions) and/or outliers, which cause traditional aggregation procedures (e.g., central tendency measures) to propose decisions with low voter satisfaction. In this research, we propose a system for the integration of crowd and expert knowledge in a crowdsourcing setting with limited resources. The system addresses the problem of sparse voting data by using machine learning models (matrix factorization and regression) for the estimation of crowd and expert votes/grades. The problem of vote aggregation under multimodal or uniform vote distributions is addressed by the inclusion of expert votes and aggregation of crowd and expert votes based on optimization and bargaining models (Kalai–Smorodinsky and Nash) usually used in game theory. Experimental evaluation on real world and artificial problems showed that the bargaining-based aggregation outperforms the traditional methods in terms of cumulative satisfaction of experts and crowd. Additionally, the machine learning models showed satisfactory predictive performance and enabled cost reduction in the process of vote collection.Keywords
Funding Information
- Office of Naval Research (ONR - N62909-19-1-2008)
This publication has 42 references indexed in Scilit:
- How social influence can undermine the wisdom of crowd effectProceedings of the National Academy of Sciences, 2011
- Crowdsourcing for relevance evaluationACM SIGIR Forum, 2008
- Intuitions About Combining Opinions: Misappreciation of the Averaging PrincipleManagement Science, 2006
- Groups of diverse problem solvers can outperform groups of high-ability problem solversProceedings of the National Academy of Sciences of the United States of America, 2004
- Stochastic gradient boostingComputational Statistics & Data Analysis, 2002
- Chapter 35 Cooperative models of bargaining1994
- The Nash bargaining solution is optimalJournal of Economic Theory, 1986
- Thirteen theorems in search of the truthTheory and Decision, 1983
- A note on aggregating opinionsOrganizational Behavior and Human Performance, 1978
- Other Solutions to Nash's Bargaining ProblemEconometrica, 1975