Roger Tourangeau
Research Subjects
Research Interests
Roger Tourangeau has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
Aaron Maitland
Senior survey methodologist at Westat, Rockville, MD, USA
3 shared publicationsHanyu Sun
Westat Inc., Statistical Department, Westat, 1600 Research Boulevard, Rockville, MD 20850, USA
3 shared publicationsFrederick G. Conrad
2 shared publications- Hanyu Sun
Westat, Rockville, MD, USA
2 shared publications Mick P. Couper
2 shared publicationsTing Yan
Westat Inc., Survey Methodology, Westat, 1600 Research Boulevard, Rockville, MD 20850, USA
2 shared publications
Latest Publications
- 21 October 2020
- Published by Oxford University Press (OUP) in Journal of Survey Statistics and Methodology
- Vol. 9 (5) , 961-991
Survey Reliability: Models, Methods, and Findings
Although most survey researchers agree that reliability is a critical requirement for survey data, there have not been many efforts to assess the reliability of responses in national surveys. In addition, there are quite different approaches to studying the reliability of survey responses. In the first section of the Lecture, I contrast a psychological theory of over-time consistency with three statistical models that use reinterview data, multi-trait multi-method experiments, and three-wave panel data to estimate reliability. The more sophisticated statistical models reflect concerns about memory effects and the impact of method factors in reinterview studies. In the following section of the Lecture, I examine some of the major findings from the literature on reliability. Despite the differences across methods for exploring reliability, the findings mostly converge, identifying similar respondent and question characteristics as major determinants of reliability. The next section of the paper looks at the correlations among estimates of reliability derived from the different methods; it finds some support for the validity of the measures from traditional reinterview studies. The empirical claims motivating the more sophisticated methods for estimating reliability are not strongly supported in the literature. Reliability is, in my judgment, a neglected topic among survey researchers, and I hope the Lecture spurs further studies of the reliability of survey questions.
- 21 October 2020
- Published by Oxford University Press (OUP) in Journal of Survey Statistics and Methodology
- Vol. 9 (5) , 961-991
Survey Reliability: Models, Methods, and Findings
Although most survey researchers agree that reliability is a critical requirement for survey data, there have not been many efforts to assess the reliability of responses in national surveys. In addition, there are quite different approaches to studying the reliability of survey responses. In the first section of the Lecture, I contrast a psychological theory of over-time consistency with three statistical models that use reinterview data, multi-trait multi-method experiments, and three-wave panel data to estimate reliability. The more sophisticated statistical models reflect concerns about memory effects and the impact of method factors in reinterview studies. In the following section of the Lecture, I examine some of the major findings from the literature on reliability. Despite the differences across methods for exploring reliability, the findings mostly converge, identifying similar respondent and question characteristics as major determinants of reliability. The next section of the paper looks at the correlations among estimates of reliability derived from the different methods; it finds some support for the validity of the measures from traditional reinterview studies. The empirical claims motivating the more sophisticated methods for estimating reliability are not strongly supported in the literature. Reliability is, in my judgment, a neglected topic among survey researchers, and I hope the Lecture spurs further studies of the reliability of survey questions.
- 8 September 2020
- Published by Oxford University Press (OUP) in Journal of Survey Statistics and Methodology
- Vol. 9 (4) , 651-673
Comparing Methods for Assessing Reliability
The usual method for assessing the reliability of survey data has been to conduct reinterviews a short interval (such as one to two weeks) after an initial interview and to use these data to estimate relatively simple statistics, such as gross difference rates (GDRs). More sophisticated approaches have also been used to estimate reliability. These include estimates from multi-trait, multi-method experiments, models applied to longitudinal data, and latent class analyses. To our knowledge, no prior study has systematically compared these different methods for assessing reliability. The Population Assessment of Tobacco and Health Reliability and Validity (PATH-RV) Study, done on a national probability sample, assessed the reliability of answers to the Wave 4 questionnaire from the PATH Study. Respondents in the PATH-RV were interviewed twice about two weeks apart. We examined whether the classic survey approach yielded different conclusions from the more sophisticated methods. We also examined two ex ante methods for assessing problems with survey questions and item nonresponse rates and response times to see how strongly these related to the different reliability estimates. We found that kappa was highly correlated with both GDRs and over-time correlations, but the latter two statistics were less highly correlated, particularly for adult respondents; estimates from longitudinal analyses of the same items in the main PATH study were also highly correlated with the traditional reliability estimates. The latent class analysis results, based on fewer items, also showed a high level of agreement with the traditional measures. The other methods and indicators had at best weak relationships with the reliability estimates derived from the reinterview data. Although the Question Understanding Aid seems to tap a different factor from the other measures, for adult respondents, it did predict item nonresponse and response latencies and thus may be a useful adjunct to the traditional measures.
- 8 September 2020
- Published by Oxford University Press (OUP) in Journal of Survey Statistics and Methodology
- Vol. 9 (4) , 651-673
Comparing Methods for Assessing Reliability
The usual method for assessing the reliability of survey data has been to conduct reinterviews a short interval (such as one to two weeks) after an initial interview and to use these data to estimate relatively simple statistics, such as gross difference rates (GDRs). More sophisticated approaches have also been used to estimate reliability. These include estimates from multi-trait, multi-method experiments, models applied to longitudinal data, and latent class analyses. To our knowledge, no prior study has systematically compared these different methods for assessing reliability. The Population Assessment of Tobacco and Health Reliability and Validity (PATH-RV) Study, done on a national probability sample, assessed the reliability of answers to the Wave 4 questionnaire from the PATH Study. Respondents in the PATH-RV were interviewed twice about two weeks apart. We examined whether the classic survey approach yielded different conclusions from the more sophisticated methods. We also examined two ex ante methods for assessing problems with survey questions and item nonresponse rates and response times to see how strongly these related to the different reliability estimates. We found that kappa was highly correlated with both GDRs and over-time correlations, but the latter two statistics were less highly correlated, particularly for adult respondents; estimates from longitudinal analyses of the same items in the main PATH study were also highly correlated with the traditional reliability estimates. The latent class analysis results, based on fewer items, also showed a high level of agreement with the traditional measures. The other methods and indicators had at best weak relationships with the reliability estimates derived from the reinterview data. Although the Question Understanding Aid seems to tap a different factor from the other measures, for adult respondents, it did predict item nonresponse and response latencies and thus may be a useful adjunct to the traditional measures.
- 1 October 2019
- Published by American Public Health Association in American Journal of Public Health
- Vol. 109 (10) , 1326-1327
Surveying Hard-to-Survey Populations Despite the Unfavorable Environment
American Journal of Public Health (AJPH) from the American Public Health Association (APHA) Surveying Hard-to-Survey Populations Despite the Unfavorable Environment, an article from American Journal of Public Health, Vol 109 Issue 10
- 1 October 2019
- Published by American Public Health Association in American Journal of Public Health
- Vol. 109 (10) , 1326-1327
Surveying Hard-to-Survey Populations Despite the Unfavorable Environment
American Journal of Public Health (AJPH) from the American Public Health Association (APHA) Surveying Hard-to-Survey Populations Despite the Unfavorable Environment, an article from American Journal of Public Health, Vol 109 Issue 10
- 24 July 2019
- Published by Oxford University Press (OUP) in Journal of Survey Statistics and Methodology
- Vol. 8 (3) , 413-432
How Errors Cumulate: Two Examples
This article examines the relationship among different types of nonobservation errors (all of which affect estimates from nonprobability internet samples) and between nonresponse and measurement errors. Both are examples of how different error sources can interact. Estimates from nonprobability samples seem to have more total error than estimates from probability samples, even ones with very low response rates. This finding suggests that the combination of coverage, selection, and nonresponse errors has greater cumulative effects than nonresponse error alone. The probabilities of having internet access, joining an internet panel, and responding to a particular survey request are probably correlated and, as a result, may lead to greater covariances with survey variables than response propensities alone; the biases accentuate one another. With nonresponse and measurement error, the two sources seem more or less uncorrelated, with one exception—those most prone to social desirability bias (those in the undesirable categories) are also less likely to respond. In addition, the propensity for unit nonresponse seems to be related to item nonresponse.
- 24 July 2019
- Published by Oxford University Press (OUP) in Journal of Survey Statistics and Methodology
- Vol. 8 (3) , 413-432
How Errors Cumulate: Two Examples
This article examines the relationship among different types of nonobservation errors (all of which affect estimates from nonprobability internet samples) and between nonresponse and measurement errors. Both are examples of how different error sources can interact. Estimates from nonprobability samples seem to have more total error than estimates from probability samples, even ones with very low response rates. This finding suggests that the combination of coverage, selection, and nonresponse errors has greater cumulative effects than nonresponse error alone. The probabilities of having internet access, joining an internet panel, and responding to a particular survey request are probably correlated and, as a result, may lead to greater covariances with survey variables than response propensities alone; the biases accentuate one another. With nonresponse and measurement error, the two sources seem more or less uncorrelated, with one exception—those most prone to social desirability bias (those in the undesirable categories) are also less likely to respond. In addition, the propensity for unit nonresponse seems to be related to item nonresponse.
- 10 December 2017
- Published by Oxford University Press (OUP) in Public Opinion Quarterly
- Vol. 81 (4) , 896-929
Web Surveys by Smartphone and Tablets
With respondents increasingly completing web surveys on tablet computers and smartphones, several studies have examined the potential effects of the switch from PCs to mobile devices. The studies have looked at a range of outcomes, including completion rates, breakoffs, and item nonresponse. We carried out a field experiment that compared responses obtained by smartphones, tablets, and laptop computers, focusing on the potential effects of the different devices on measurement errors. We examined whether the differences across devices in screen size (and the related need to scroll to see the entire question or the full set of response options) might moderate the effects of response order, affect the strategy respondents used to decide which of two options was preferable, change the effect of question context, or influence the use of definitions. Our experiments were based on the principle of visual prominence—the idea that respondents are more likely to notice and consider information that is easy to see. The experiments were deliberately designed to maximize the impact of screen size on the results, since the screen size would affect the visual prominence of key information. However, like many of the prior studies examining mobile devices, although response order, context, and evaluation strategy affected the answers respondents gave, few device effects emerged.
- 10 December 2017
- Published by Oxford University Press (OUP) in Public Opinion Quarterly
- Vol. 81 (4) , 896-929
Web Surveys by Smartphone and Tablets
With respondents increasingly completing web surveys on tablet computers and smartphones, several studies have examined the potential effects of the switch from PCs to mobile devices. The studies have looked at a range of outcomes, including completion rates, breakoffs, and item nonresponse. We carried out a field experiment that compared responses obtained by smartphones, tablets, and laptop computers, focusing on the potential effects of the different devices on measurement errors. We examined whether the differences across devices in screen size (and the related need to scroll to see the entire question or the full set of response options) might moderate the effects of response order, affect the strategy respondents used to decide which of two options was preferable, change the effect of question context, or influence the use of definitions. Our experiments were based on the principle of visual prominence—the idea that respondents are more likely to notice and consider information that is easy to see. The experiments were deliberately designed to maximize the impact of screen size on the results, since the screen size would affect the visual prominence of key information. However, like many of the prior studies examining mobile devices, although response order, context, and evaluation strategy affected the answers respondents gave, few device effects emerged.
- 23 October 2017
- Published by Springer Nature
- p. 135-141
Maintaining Respondent Trust and Protecting Their Data
- 23 October 2017
- Published by Springer Nature
- p. 135-141
Maintaining Respondent Trust and Protecting Their Data
- Open Access
- 1 September 2017
- Published by SAGE Publications in Journal of Official Statistics
- Vol. 33 (3) , 735-752
Responsive Survey Designs for Reducing Nonresponse Bias
Survey researchers have been investigating alternative approaches to reduce data collection costs while mitigating the risk of nonresponse bias or to produce more accurate estimates within the same budget. Responsive or adaptive design has been suggested as one means for doing this. Falling survey response rates and the need to find effective ways of implementing responsive design has focused attention on the relationship between response rates and nonresponse bias. In our article, we re-examine the data compiled by Groves and Peytcheva (2008) in their influential article and show there is an important between-study component of variance in addition to the within-study variance highlighted in the original analysis. We also show that theory implies that raising response rates can help reduce the nonresponse bias on average across the estimates within a study. We then propose a typology of response propensity models that help explain the empirical findings, including the relative weak relationship between nonresponse rates and nonresponse bias. Using these results, we explore when responsive design tools such as switching modes, giving monetary incentives, and increasing the level of effort are likely to be effective. We conclude with some comments on the use of responsive design and weighting to control nonresponse bias.
- 1 September 2017
- Published by SAGE Publications in Journal of Official Statistics
- Vol. 33 (3) , 735-752
Open AccessResponsive Survey Designs for Reducing Nonresponse Bias
Survey researchers have been investigating alternative approaches to reduce data collection costs while mitigating the risk of nonresponse bias or to produce more accurate estimates within the same budget. Responsive or adaptive design has been suggested as one means for doing this. Falling survey response rates and the need to find effective ways of implementing responsive design has focused attention on the relationship between response rates and nonresponse bias. In our article, we re-examine the data compiled by Groves and Peytcheva (2008) in their influential article and show there is an important between-study component of variance in addition to the within-study variance highlighted in the original analysis. We also show that theory implies that raising response rates can help reduce the nonresponse bias on average across the estimates within a study. We then propose a typology of response propensity models that help explain the empirical findings, including the relative weak relationship between nonresponse rates and nonresponse bias. Using these results, we explore when responsive design tools such as switching modes, giving monetary incentives, and increasing the level of effort are likely to be effective. We conclude with some comments on the use of responsive design and weighting to control nonresponse bias.