Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study
Top Cited Papers
Open Access
- 23 September 2021
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 16 (9) , e0257535
Abstract
Effect sizes are commonly interpreted using heuristics established by Cohen (e.g., small: r = .1, medium r = .3, large r = .5), despite mounting evidence that these guidelines are mis-calibrated to the effects typically found in psychological research. This study’s aims were to 1) describe the distribution of effect sizes across multiple instruments, 2) consider factors qualifying the effect size distribution, and 3) identify examples as benchmarks for various effect sizes. For aim one, effect size distributions were illustrated from a large, diverse sample of 9/10-year-old children. This was done by conducting Pearson’s correlations among 161 variables representing constructs from all questionnaires and tasks from the Adolescent Brain and Cognitive Development Study® baseline data. To achieve aim two, factors qualifying this distribution were tested by comparing the distributions of effect size among various modifications of the aim one analyses. These modified analytic strategies included comparisons of effect size distributions for different types of variables, for analyses using statistical thresholds, and for analyses using several covariate strategies. In aim one analyses, the median in-sample effect size was .03, and values at the first and third quartiles were .01 and .07. In aim two analyses, effects were smaller for associations across instruments, content domains, and reporters, as well as when covarying for sociodemographic factors. Effect sizes were larger when thresholding for statistical significance. In analyses intended to mimic conditions used in “real-world” analysis of ABCD data, the median in-sample effect size was .05, and values at the first and third quartiles were .03 and .09. To achieve aim three, examples for varying effect sizes are reported from the ABCD dataset as benchmarks for future work in the dataset. In summary, this report finds that empirically determined effect sizes from a notably large dataset are smaller than would be expected based on existing heuristics.Keywords
Funding Information
- NIH/NIDA (T32DA043593)
This publication has 18 references indexed in Scilit:
- The N-Pact Factor: Evaluating the Quality of Empirical Journals with Respect to Sample Size and Statistical PowerPLOS ONE, 2014
- Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample SizePLOS ONE, 2014
- Erratum: Power failure: why small sample size undermines the reliability of neuroscienceNature Reviews Neuroscience, 2013
- Power failure: why small sample size undermines the reliability of neuroscienceNature Reviews Neuroscience, 2013
- On effect size.Psychological Methods, 2012
- The Biopsychosocial Model 25 Years Later: Principles, Practice, and Scientific InquiryThe Annals of Family Medicine, 2004
- The Five Factor Model and impulsivity: using a structural model of personality to understand impulsivityPersonality and Individual Differences, 2001
- Psychological testing and psychological assessment: A review of evidence and issues.American Psychologist, 2001
- The Child Behavior Checklist and Related Forms for Assessing Behavioral/Emotional Problems and CompetenciesPediatrics in Review, 2000
- The Need for a New Medical Model: A Challenge for BiomedicineScience, 1977