PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS
- 10 November 2006
- journal article
- Published by Wiley in Personnel Psychology
- Vol. 59 (4) , 927-953
Abstract
This article has 2 goals. First, we discuss publication bias and explain why it presents a potential problem for industrial and organizational psychology. After reviewing the traditional failsafe N, or file drawer analysis, we introduce a more sophisticated method of publication bias analysis (trim and fill), which has been developed in the medical literature but is largely unfamiliar to industrial and organizational psychology researchers. Second, we demonstrate trim and fill by applying it to validity information reported in the technical manuals of 4 test vendors. In doing so, we assess the likelihood that criterion‐related validity information provided by test publishers may overestimate test validity. In our analysis of 18 validity distributions, we found evidence of either no or minimal bias for 2 of the vendors' distributions and evidence of moderate‐to‐severe bias in at least 1 distribution from each of the other 2 vendors. In both cases in which publication bias was found, we noted instances in which the publishers tended to report only statistically significant correlations and that this practice was detected using publication bias methodology.Keywords
This publication has 48 references indexed in Scilit:
- Adjusting for publication bias in the presence of heterogeneityStatistics in Medicine, 2003
- A Meta-Analytic Study of General Mental Ability Validity for Different Occupations in the European Community.Journal of Applied Psychology, 2003
- Perceived organizational support: A review of the literature.Journal of Applied Psychology, 2002
- Perceived organizational support: A review of the literature.Journal of Applied Psychology, 2002
- A Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-AnalysisJournal of the American Statistical Association, 2000
- Validity of Customer Service Measures in Personnel Selection: A Review of Criterion and Construct EvidenceHuman Performance, 1998
- Integrity tests: Facts and unresolved issues.American Psychologist, 1994
- A meta-analytic review of the relationship between absence and turnover.Journal of Applied Psychology, 1992
- The file drawer problem and tolerance for null results.Psychological Bulletin, 1979
- Consequences of prejudice against the null hypothesis.Psychological Bulletin, 1975