Elicitation and aggregation of multimodal estimates improve wisdom of crowd effects on ordering tasks
Open Access
- 1 February 2024
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 14 (1) , 1-13
Abstract
We present a wisdom of crowds study where participants are asked to order a small set of images based on the number of dots they contain and then to guess the respective number of dots in each image. We test two input elicitation interfaces—one elicits the two modalities of estimates jointly and the other independently. We show that the latter interface yields higher quality estimates, even though the multimodal estimates tend to be more self-contradictory. The inputs are aggregated via optimization and voting-rule based methods to estimate the true ordering of a larger universal set of images. We demonstrate that the quality of collective estimates from the simpler yet more computationally-efficient voting methods is comparable to that achieved by the more complex optimization model. Lastly, we find that using multiple modalities of estimates from one group yields better collective estimates compared to mixing numerical estimates from one group with the ordinal estimates from a different group.Keywords
Funding Information
- National Science Foundation (Award 1850355)
This publication has 37 references indexed in Scilit:
- The Dot-Guessing Game: A ‘Fruit Fly’ for Human Computation ResearchSSRN Electronic Journal, 2010
- Smaller is better (when sampling from the crowd within): Low memory-span individuals benefit more from multiple opportunities for estimation.Journal of Experimental Psychology: Learning, Memory, and Cognition, 2010
- The Wisdom of Many in One MindPsychological Science, 2009
- Forecasting ElectionsPublic Opinion Quarterly, 2009
- Measuring the Crowd WithinPsychological Science, 2008
- Overconfidence in Interval Estimates.Journal of Experimental Psychology: Learning, Memory, and Cognition, 2004
- The effects of averaging subjective probability estimates between and within judges.Journal of Experimental Psychology: Applied, 2000
- The empirical case for two systems of reasoning.Psychological Bulletin, 1996
- Multiple explanation: A consider-an-alternative strategy for debiasing judgments.Journal of Personality and Social Psychology, 1995
- Ranking or rating? Some data and their implications for the measurement of evaluative responseBritish Journal of Psychology, 1994