It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks
Open Access
- 1 April 2015
- journal article
- research article
- Published by SAGE Publications in Sage Open
- Vol. 5 (2)
Abstract
Instructional manipulation checks (IMCs) have become popular tools for identifying inattentive participants in online studies. IMCs function by attempting to trick inattentive participants into responding incorrectly. However, from a conversational perspective, question characteristics are part of the researcher’s contribution to the conversation, and IMCs may teach participants that there is “more than meets the eye,” prompting systematic thinking on subsequent tricky-seeming questions in an attempt to avoid being tricked. In two online studies, participants responded to a simple task either before or after completing an IMC. As expected, answering an IMC prior to the task improved performance on items that benefit from increased systematic thinking—namely, the Cognitive Reflection Test (Study 1), and a probabilistic reasoning task (Study 2). We conclude that IMCs change attention rather than merely measure attention and discuss implications for their use in online studies.Keywords
This publication has 19 references indexed in Scilit:
- Are ‘counter-intuitive’ deontological judgments really counter-intuitive? An empirical reply to Kahaneet al.(2012)Social Cognitive and Affective Neuroscience, 2013
- Reflection and Reasoning in Moral JudgmentCognitive Science, 2011
- The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasksMemory & Cognition, 2011
- Philosophy's New Challenge: Experiments and Intentional ActionMind & Language, 2011
- Instructional manipulation checks: Detecting satisficing to increase statistical powerJournal of Experimental Social Psychology, 2009
- Cognitive Reflection and Decision MakingJournal of Economic Perspectives, 2005
- SURVEY RESEARCHAnnual Review of Psychology, 1999
- Self-reports: How the questions shape the answers.American Psychologist, 1999
- Judgment in a Social Context: Biases, Shortcomings, and the Logic of Conversation1994
- Conflict between intuitive and rational processing: When people behave against their better judgment.Journal of Personality and Social Psychology, 1994