Distinction without a difference? An assessment of MTurk Worker types
Authors:
Open Access
- 28 January 2020
- journal article
- research article
- Published by SAGE Publications in Research & Politics
- Vol. 7 (1)
Abstract
Amazon’s Mechanical Turk (MTurk) platform is a popular tool for scholars seeking a reasonably representative population to recruit subjects for academic research that is cheaper than contract work via survey research firms. Numerous scholarly inquiries affirm that the MTurk pool is at least as representative as college student samples; however, questions about the validity of MTurk data persist. Amazon classifies all MTurk Workers into two types: (1) “regular” Workers, and (2) more qualified (and expensive) “master” Workers. In this paper, we evaluate how choice in Worker type impacts the nature of research samples in terms of characteristics/features and performance. Our results identify few meaningful differences between master and regular Workers. However, we do find that master Workers are more likely to be female, older, and Republican, than regular Workers. Additionally, master Workers have far more experience, having spent twice as much time working on MTurk and having completed over seven times the number of assignments. Based on these findings, we recommend that researchers ask for Worker status and number of assignments completed to control for effects related to experience. However, the results imply that budget-conscious scholars will not compromise project integrity by using the wider pool of regular Workers in academic studies.Keywords
This publication has 21 references indexed in Scilit:
- Distrust and the positive test heuristic: Dispositional and situated social distrust improves performance on the Wason Rule Discovery Task.Journal of Experimental Psychology: General, 2014
- Reputation as a sufficient condition for data quality on Amazon Mechanical TurkBehavior Research Methods, 2013
- Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchersBehavior Research Methods, 2013
- Conducting behavioral research on Amazon’s Mechanical TurkBehavior Research Methods, 2011
- Developing a tool for measuring the decision-making competence of older adults.Psychology and Aging, 2010
- Cognitive Reflection and Decision MakingJournal of Economic Perspectives, 2005
- Practice Effects on Commonly Used Measures of Executive Function Across Twelve MonthsThe Clinical Neuropsychologist, 1999
- Media Framing of a Civil Liberties Conflict and Its Effect on ToleranceAmerican Political Science Review, 1997
- The Role of Partisan Stereotypes in Information Processing about Political CandidatesAmerican Journal of Political Science, 1993
- Choices, values, and frames.American Psychologist, 1984