Are Survey Experiments Externally Valid?
Top Cited Papers
- 7 May 2010
- journal article
- research article
- Published by Cambridge University Press (CUP) in American Political Science Review
- Vol. 104 (2) , 226-242
Abstract
Researchers use survey experiments to establish causal effects in descriptively representative samples, but concerns remain regarding the strength of the stimuli and the lack of realism in experimental settings. We explore these issues by comparing three national survey experiments on Medicare and immigration with contemporaneous natural experiments on the same topics. The survey experiments reveal that providing information increases political knowledge and alters attitudes. In contrast, two real-world government announcements had no discernable effects, except among people who were exposed to the same facts publicized in the mass media. Even among this exposed subsample, treatment effects were smaller and sometimes pointed in the opposite direction. Methodologically, our results suggest the need for caution when extrapolating from survey experiments. Substantively, we find that many citizens are able to recall factual information appearing in the news but may not adjust their beliefs and opinions in response to this information.Keywords
This publication has 62 references indexed in Scilit:
- Observing the Counterfactual? The Search for Political Experiments in NaturePolitical Analysis, 2009
- Identifying the Persuasive Effects of Presidential AdvertisingAmerican Journal of Political Science, 2007
- Neighborhood Information Exchange and Voter Participation: An Experimental StudyAmerican Political Science Review, 2006
- The Growth and Development of Experimental Research in Political ScienceAmerican Political Science Review, 2006
- Who Evaluates a Presidential Candidate by Using Non-Policy Campaign Messages?Political Research Quarterly, 2005
- Social Security and MedicarePublic Opinion Quarterly, 2004
- Political Preference Formation: Competition, Deliberation, and the (Ir)relevance of Framing EffectsAmerican Political Science Review, 2004
- Considered Opinions: Deliberative Polling in BritainBritish Journal of Political Science, 2002
- Misinformation and the Currency of Democratic CitizenshipThe Journal of Politics, 2000
- Sampling Weights and Regression AnalysisSociological Methods & Research, 1994