Demand Effects in Survey Experiments: An Empirical Assessment
Top Cited Papers
- 11 December 2018
- journal article
- research article
- Published by Cambridge University Press (CUP) in American Political Science Review
- Vol. 113 (2) , 517-529
Abstract
Survey experiments are ubiquitous in social science. A frequent critique is that positive results in these studies stem from experimenter demand effects (EDEs)—bias that occurs when participants infer the purpose of an experiment and respond so as to help confirm a researcher’s hypothesis. We argue that online survey experiments have several features that make them robust to EDEs, and test for their presence in studies that involve over 12,000 participants and replicate five experimental designs touching on all empirical political science subfields. We randomly assign participants information about experimenter intent and show that providing this information does not alter the treatment effects in these experiments. Even financial incentives to respond in line with researcher expectations fail to consistently induce demand effects. Research participants exhibit a limited ability to adjust their behavior to align with researcher expectations, a finding with important implications for the design and interpretation of survey experiments.Keywords
This publication has 48 references indexed in Scilit:
- Sources of Bias in Retrospective Decision Making: Experimental Evidence on Voters’ Limitations in Controlling IncumbentsAmerican Political Science Review, 2012
- Elite Influence on Public Opinion in an Informed ElectorateAmerican Political Science Review, 2011
- Experimenter demand effects in economic experimentsExperimental Economics, 2010
- The Logic of the Survey Experiment ReexaminedPolitical Analysis, 2007
- Deception in Psychology: Moral Costs and Benefits of Unsought Self-KnowledgeAccountability in Research, 2006
- Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market DiscriminationAmerican Economic Review, 2004
- The Consquences of Adjustment for a Concomitant Variable That Has Been Affected by the TreatmentJournal of the Royal Statistical Society Series A: Statistics in Society, 1984
- Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference.Psychological Bulletin, 1972
- Demand characteristics and three conceptions of the frequently deceived subject.Journal of Personality and Social Psychology, 1970
- On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications.American Psychologist, 1962