The Danger of Testing by Selecting Controlled Subsets, with Applications to Spoken-Word Recognition
Open Access
- 24 January 2019
- journal article
- Published by Ubiquity Press, Ltd. in Journal of Cognition
- Vol. 2 (1) , 2
Abstract
When examining the effects of a continuous variable x on an outcome y, a researcher might choose to dichotomize on x, dividing the population into two sets-low x and high x-and testing whether these two subpopulations differ with respect to y. Dichotomization has long been known to incur a cost in statistical power, but there remain circumstances in which it is appealing: an experimenter might use it to control for confounding covariates through subset selection, by carefully choosing a subpopulation of Low and a corresponding subpopulation of High that are balanced with respect to a list of control variables, and then comparing the subpopulations' y values. This "divide, select, and test" approach is used in many papers throughout the psycholinguistics literature, and elsewhere. Here we show that, despite the apparent innocuousness, these methodological choices can lead to erroneous results, in two ways. First, if the balanced subsets of Low and High are selected in certain ways, it is possible to conclude a relationship between x and y not present in the full population. Specifically, we show that previously published conclusions drawn from this methodology-about the effect of a particular lexical property on spoken-word recognition-do not in fact appear to hold. Second, if the balanced subsets of Low and High are selected randomly, this methodology frequently fails to show a relationship between x and y that is present in the full population. Our work uncovers a new facet of an ongoing research effort: to identify and reveal the implicit freedoms of experimental design that can lead to false conclusions.Keywords
This publication has 60 references indexed in Scilit:
- Sizing up the competition: Quantifying the influence of the mental lexicon on auditory and visual spoken word recognitionThe Journal of the Acoustical Society of America, 2011
- An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational StudiesMultivariate Behavioral Research, 2011
- Statistical issues in the use of dynamic allocation methods for balancing baseline covariatesBritish Journal of Cancer, 2011
- Emotion's influence on memory for spatial and temporal contextCognition and Emotion, 2011
- Clustering coefficients of lexical neighborhoodsThe Mental Lexicon, 2010
- The influence of the phonological neighborhood clustering coefficient on spoken word recognition.Journal of Experimental Psychology: Human Perception and Performance, 2009
- Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed modelsJournal of Memory and Language, 2008
- Mere Exposure and Racial Prejudice: Exposure to Other-Race Faces Increases Liking for Strangers of that RaceSocial Cognition, 2008
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- The central role of the propensity score in observational studies for causal effectsBiometrika, 1983