Aaron J. Moss

Researchers who employ online data collection from human subjects currently face a conundrum: It is both essential to how behavioral science functions and threatened by low-quality data. It is often assumed that random, inconsistent, and otherwise incomprehensible data in online surveys comes mainly from bots. Despite this assumption, few studies have directly examined where problematic data comes from, even though identifying the source has important implications for creating the right solutions. We examined this issue on several popular participant-recruitment platforms, including Mechanical Turk (MTurk) and Lucid. Across four studies spanning 5 years using multiple methods, we here provide evidence that most of the data-quality problems affecting online research using online panels can be tied to fraudulent users from outside of the United States—not bots. We identify many of the telltale signs that humans leave behind and describe the most effective ways of blocking problematic human responses to address the online data-quality problem.

It is important for researchers to be able to identify sources of low-quality data in their surveys. At the same time, it is important that the methods researchers use to detect these sources of low-quality data do not result in a substantially worse survey-taking experience for their participants. The purpose of the present project was to investigate how two types of attention check items—infrequency/frequency items (e.g., “I don’t have a first name”) and instructed-response items (e.g., “Choose disagree for this item”)—impact the survey-taking experiences of participants. In Study 1 (N = 400), we found that participants responded most favourably to a survey composed entirely of control items, followed by a survey composed entirely of infrequency/frequency items and, finally, a survey composed entirely of instructed-response items. However, in Study 2 (N = 793) and Study 3 (N = 800), we found that these effects only appeared when over half of the survey consisted of infrequency/frequency items and instructed-response items, respectively. These results suggest that attention check items can negatively impact a participant’s survey-taking experience but only when administered in large quantities. They also suggest that infrequency/frequency items tend to be better received than instructed-response items.

Online data collection from human subjects currently faces a conundrum: it is both essential to how behavioral science functions and threatened by low-quality data. It is often assumed that random, inconsistent, and otherwise incomprehensible data in online surveys comes mainly from “bots.” Despite this assumption, few studies have directly examined where problematic data comes from, even though identifying the source has important implications for creating the right solutions. We examined this issue on several popular participant recruitment platforms, including Mechanical Turk and Lucid. Across four studies spanning five years using multiple methods, we provide evidence that most of the data quality problems affecting online research using online panels can be tied to fraudulent users from outside of the US—not bots. We identify many of the telltale signs that humans leave behind and describe the most effective ways of blocking problematic human responses to address the online data quality problem.

This paper introduces Connect, CloudResearch's innovative platform designed to revolutionize the realm of online participant recruitment in social and behavioral science research. Operating as a marketplace, Connect facilitates interactions between researchers and participants, enabling the deployment of surveys and experiments constructed via third-party tools such as Qualtrics, SurveyMonkey, and Google forms. With its current focus on the U.S. demographic (with plans of future expansion to other English-speaking countries) and those aged 18 and above, Connect proves versatile in accommodating a diverse range of studies, including academic, market, and user experience research, as well as machine learning. Connect's uniqueness lies in its tri-fold emphasis on advanced features, data quality, and affordability. Advanced attributes include collaborative tools, a flexible API, and capabilities supporting intricate study designs. To assure impeccable data quality, Connect incorporates Sentry®, CloudResearch’s proprietary participant vetting mechanism, coupled with stringent technical evaluations. Despite these advancements, Connect remains cost-effective, charging researchers the lowest service fee in the online recruitment sector. This paper delves deeper into the platform’s attributes, with particular attention to its emphasis on participant experience, advanced functionalities, and the robust data quality assurance methods in place.

To understand human behavior, social scientists need people and data. In the last decade, Amazon’s Mechanical Turk (MTurk) emerged as a flexible, aff

Behavioral scientists looking to run online studies are confronted with a bevy of options. Where to recruit participants? Which tools to use for survey creation and study management? How to maintain data quality? In this tutorial, we highlight the unique capabilities of market-research panels and demonstrate how researchers can effectively sample from such panels. Unlike the microtask platforms most academics are familiar with (e.g., MTurk and Prolific), market-research panels have access to more than 100 million potential participants worldwide, provide more representative samples, and excel at demographic targeting. However, efficiently gathering data from online panels requires integration between the panel and a researcher’s survey in ways that are uncommon on microtask sites. For example, panels allow researchers to target participants according to preprofiled demographics (“Level 1” targeting, e.g., parents) and demographics that are not preprofiled but are screened for within the survey (“Level 2” targeting, e.g., parents of autistic children). In this article, we demonstrate how to sample hard-to-reach groups using market-research panels. We also describe several best practices for conducting research using online panels, including setting in-survey quotas to control sample composition and managing data quality. Our aim is to provide researchers with enough information to determine whether market-research panels are right for their research and to outline the necessary considerations for using such panels.

Behavioral scientists looking to run online studies are confronted with a bevy of options. Where to recruit participants? Which tools to use for survey creation and study management? How to maintain data quality? In this tutorial, we highlight the unique capabilities of market research panels and demonstrate how researchers can effectively sample from such panels. Unlike the microtask platforms most academics are familiar with (e.g., MTurk and Prolific), market research panels have access to over 100 million potential participants worldwide, provide more representative samples, and excel at demographic targeting. However, efficiently gathering data from online panels requires integration between the panel and a researcher’s survey in ways that are uncommon on microtask sites. For example, panels allow researchers to target participants based on pre-profiled demographics (‘Level 1’ targeting, e.g., parents) and based on demographics that are not pre-profiled but screened for within the survey (‘Level 2’ targeting, e.g., parents of autistic children). Our article demonstrates how to sample hard-to-reach groups using market research panels. We also describe several best practices for conducting research using online panels, including setting in-survey quotas to control sample composition and managing data quality. Our aim is to provide researchers with enough information to determine whether market research panels are right for their research and to outline the necessary considerations for using such panels.

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk’s Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. CloudResearch, a third-party website that interfaces with MTurk, assessed ~100,000 MTurkers and categorized them into those that provide high- (~65,000, Approved) and low-(~35,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch’s vetting. Participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample, completed an array of data quality measures. Approved participants had better reading comprehension, reliability, honesty, and attentiveness scores, were less likely to cheat and satisfice, and replicated classic experimental effects more reliably than Blocked participants who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss the implications of using the Approved group for scientific studies conducted on Mechanical Turk.