Disentangling Bias and Variance in Election Polls
Authors:
- 3 April 2018
- journal article
- research article
- Published by Taylor & Francis in Journal of the American Statistical Association
- Vol. 113 (522) , 607-614
Abstract
It is well known among researchers and practitioners that election polls suffer from a variety of sampling and nonsampling errors, often collectively referred to as total survey error. Reported margins of error typically only capture sampling variability, and in particular, generally ignore nonsampling errors in defining the target population (e.g., errors due to uncertainty in who will vote). Here, we empirically analyze 4221 polls for 608 state-level presidential, senatorial, and gubernatorial elections between 1998 and 2014, all of which were conducted during the final three weeks of the campaigns. Comparing to the actual election outcomes, we find that average survey error as measured by root mean square error is approximately 3.5 percentage points, about twice as large as that implied by most reported margins of error. We decompose survey error into election-level bias and variance terms. We find that average absolute election-level bias is about 2 percentage points, indicating that polls for a given election often share a common component of error. This shared error may stem from the fact that polling organizations often face similar difficulties in reaching various subgroups of the population, and that they rely on similar screening rules when estimating who will vote. We also find that average election-level variance is higher than implied by simple random sampling, in part because polling organizations often use complex sampling designs and adjustment procedures. We conclude by discussing how these results help explain polling failures in the 2016 U.S. presidential election, and offer recommendations to improve polling practice.Keywords
Funding Information
- Division of Mathematical Sciences (DMS-1308566,DMS-1309507,DMS-1418172,DMS-1554804)
- National Natural Science Foundation of China (11571009)
This publication has 8 references indexed in Scilit:
- The Mythical Swing VoterQuarterly Journal of Political Science, 2016
- Total Survey Error: Past, Present, and FuturePublic Opinion Quarterly, 2010
- Total Survey Error: Design, Implementation, and EvaluationPublic Opinion Quarterly, 2010
- Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper)Bayesian Analysis, 2006
- A Review: Preelection Survey Methodology: Details From Eight Polling Organizations, 1988 and 1992Public Opinion Quarterly, 1995
- Poll FaultingCHANCE, 1993
- That Which We Call Welfare by Any Other Name Would Smell Sweeter an Analysis of the Impact of Question Wording on Response PatternsPublic Opinion Quarterly, 1987
- Effects of Question Order on Survey ResponsesPublic Opinion Quarterly, 1981