Achieving Statistical Significance with Control Variables and Without Transparency
- 16 November 2020
- journal article
- research article
- Published by Cambridge University Press (CUP) in Political Analysis
- Vol. 29 (3) , 356-369
Abstract
How often do articles depend on suppression effects for their findings? How often do they disclose this fact? By suppression effects, we mean control-variable-induced increases in estimated effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that authors have a strong explanation for them, especially when the statistical significance of the key finding depends on them. In a reanalysis of observational studies from a leading journal, we find that over 30% of articles depend on suppression effects for statistical significance. Although increases in key effect estimates from including control variables are of course potentially justifiable, none of the articles justify or disclose them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.Keywords
This publication has 41 references indexed in Scilit:
- Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan*The Quarterly Journal of Economics, 2012
- Invited Commentary: Understanding Bias AmplificationAmerican Journal of Epidemiology, 2011
- Illustrating bias due to conditioning on a colliderInternational Journal of Epidemiology, 2009
- The Phantom Menace: Omitted Variable Bias in Econometric ResearchConflict Management and Peace Science, 2005
- Why Most Published Research Findings Are FalsePLOS Medicine, 2005
- TOWARD A NEW POLITICAL METHODOLOGY: Microfoundations and ARTAnnual Review of Political Science, 2002
- Suppression situations in psychological research: Definitions, implications, and applications.Psychological Bulletin, 1991
- Suppression and Enhancement in Bivariate RegressionJournal of the Royal Statistical Society: Series D (The Statistician), 1986
- Suppressor Structures in the General Linear ModelEducational and Psychological Measurement, 1983
- A Revised Definition for Suppressor Variables: a Guide To Their Identification and InterpretationEducational and Psychological Measurement, 1974