Survey Professionalism: New Evidence from Web Browsing Data
Open Access
- 6 October 2025
- journal article
- research article
- Published by Cambridge University Press (CUP) in Political Analysis
- Vol. 34 (3) , 432-450
Abstract
Online panels have become an important resource for research in political science, but the compensation offered to panelists incentivizes them to become “survey professionals,” raising concerns about data quality. We provide evidence on survey professionalism exploring three US samples of subjects who donated their browsing data, recruited via Lucid, YouGov, and Facebook (total n equals 3 comma 886 $n = 3,886$ ). Survey professionalism is common, but varies across samples: by our most conservative estimate, we find 1.7% of respondents on Facebook, 7. backslash color b l a c k Baseline 6 $\color {black}6$ % on YouGov, and 34 backslash color b l a c k period 7 $\color {black}.7$ % on Lucid to be professionals (under the assumption that professionals are as likely as non-professionals to donate data after conditioning on observable demographics available from all online survey takers). However, evidence that professionals lower data quality is limited: they do not systematically differ demographically or politically from non-professionals and do not exhibit more response instability. They are, however, somewhat more likely to speed, straightline, and attempt to take questionnaires repeatedly. To address potential selection issues in donating of browsing data, we present sensitivity analyses with lower bounds for survey professionalism. While concerns about professionalism are warranted, we conclude that survey professionals do not, by and large, distort inferences of research based on online panels.
Keywords
All Related Versions
Funding Information
- European Research Council (ERC Starting Grant EXPO-756301, Magdalena Wojciesz)
- New York University
This publication has 31 references indexed in Scilit:
- The micro-task market for lemons: data quality on Amazon's Mechanical TurkPolitical Science Research and Methods, 2021
- A systematic review and meta-analysis of discrepancies between logged and self-reported digital media useNature Human Behaviour, 2021
- Convenience Samples in Political Science Experiments2021
- Internet Panels, Professional Respondents, and Data QualityMethodology, 2015
- Causal Inference for Statistics, Social, and Biomedical Sciences2015
- Detecting Insufficient Effort Responding with an Infrequency Scale: Evaluating Validity and Participant ReactionsJournal of Business and Psychology, 2014
- A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples12014
- Assessing the reliability of the M5-120 on Amazon’s mechanical TurkComputers in Human Behavior, 2013
- Polarization and Issue Consistency Over TimePolitical Behavior, 2010
- Generalized Raking Procedures in Survey SamplingJournal of the American Statistical Association, 1993