Flexibility in reaction time analysis: many roads to a false positive?
Open Access
- 5 February 2020
- journal article
- research article
- Published by The Royal Society in Royal Society Open Science
- Vol. 7 (2) , 190831
Abstract
In the present article, we explore the influence of undisclosed flexibility in the analysis of reaction times (RTs). RTs entail some degrees of freedom of their own, due to their skewed distribution, the potential presence of outliers and the availability of different methods to deal with these issues. Moreover, these degrees of freedom are usually not considered part of the analysis itself, but preprocessing steps that are contingent on data. We analysed the impact of these degrees of freedom on the false-positive rate using simulations over real and simulated data. When several preprocessing methods are used in combination, the false-positive rate can easily rise to 17%. This figure becomes more concerning if we consider that more degrees of freedom are awaiting down the analysis pipeline, potentially making the final false-positive rate much higher.Keywords
All Related Versions
Funding Information
- Ministerio de Economía y Competitividad (PSI2017-85159-P)
- European Regional Development Fund (PSI2016-75558-P)
- Consejería de Educación, Juventud y Deporte, Comunidad de Madrid (2016-T1/SOC-1395)
- Agència de Gestió d'Ajuts Universitaris i de Recerca (2017 SGR 1545)
This publication has 28 references indexed in Scilit:
- The French Lexicon Project: Lexical decision data for 38,840 French words and 38,840 pseudowordsBehavior Research Methods, 2010
- Permutation Methods: A Basis for Exact InferenceStatistical Science, 2004
- Short Report: Reaction Time Analysis with Outlier Exclusion: Bias Varies with Sample SizeThe Quarterly Journal of Experimental Psychology Section A, 1991
- Analysis of response time distributions: An example using the Stroop task.Psychological Bulletin, 1991
- A warning about median reaction time.Journal of Experimental Psychology: Human Perception and Performance, 1988
- A warning about median reaction time.Journal of Experimental Psychology: Human Perception and Performance, 1988
- Analysis of response time distributions in the study of cognitive processes.Journal of Experimental Psychology: Learning, Memory, and Cognition, 1984
- Group reaction time distributions and an analysis of distribution statistics.Psychological Bulletin, 1979
- Retrieval processes in recognition memory.Psychological Review, 1976
- Inferred components of reaction times as functions of foreperiod duration.Journal of Experimental Psychology, 1965