Douglas L. Weed

Reviews are critical to the theory and practice of oncology. Reviews not only summarize large bodies of information, but also inform medical and public health decision-making and guide policy and research priorities (1–7). Reviews enhance transparency and increase efficiency in the scientific process (8). The purpose of this Commentary is to encourage submissions of systematic reviews (with or without meta-analyses) to the Journal. In addition, I will describe the reasons why systematic reviews are superior to so-called “narrative” reviews, provide guidelines for prospective authors of systematic reviews, and provide guidelines for evaluating the quality of any published review, systematic or not.

Now that two groups have challenged the idea that the potential outcomes (counterfactual) approach (POA) is the only legitimate approach to causal inference in epidemiology, I fully expect the supporters of that view to either defend their position or offer an olive branch to appease their critics. The debate is engaged and best left to those in academic towers where these arguments mean something. In the practice of epidemiology, despite what Vandenbroucke et al.¹ and Krieger and Davey Smith² contend, the presumed negative impact of POA has yet to be realized. No journal requires it. Sometimes counterfactuals and DAGs appear and often they do not. In the complex and ever-expanding worlds of regulation and litigation, where epidemiology plays a key role, POA is at best a whisper. Simply put, the debate about whether POA is the only legitimate approach to causal inference in epidemiology is as much about the power of individuals at certain academic institutions to gain attention as it is about the intellectual competitions that excite so-called ‘theoreticians’ of epidemiology. It remains to be seen if the exclusivity claim is really a critical issue for the future of epidemiology.

Assessment of design heterogeneity conducted prior to meta-analysis is infrequently reported; it is often presented post hoc to explain statistical heterogeneity. However, design heterogeneity determines the mix of included studies and how they are analyzed in a meta-analysis, which in turn can importantly influence the results. The goal of this work is to introduce ways to improve the assessment and reporting of design heterogeneity prior to statistical summarization of epidemiologic studies.

The methodological quality of published reviews of nutrition and cancer (2008–2009) and of the carcinogenicity of acrylamide (1999–2009) was systematically assessed. Each review was examined with respect to four characteristics: whether the purpose of the review was explicitly stated, whether a methods section (detailing the methods used to “weigh” the evidence) was included, whether “weight of evidence” methods were described elsewhere in the paper (e.g., in the discussion), and finally, whether references to recognized “weight of evidence” methods were included. In this study, ninety per cent of a systematically selected sample of recent reviews on nutrition and cancer published in 2008–2009 and 74% of reviews on acrylamide on cancer published in 1999–2009 were found to be methodologically troublesome or frankly unsound. Failure of peer review and editorial oversight are possible explanations, suggesting a broad lack of concern about this issue in the scientific community. If peer reviewers in the nutrition and cancer community do not require “weight of evidence” methods, then these methods may not appear in the published reviews. Similarly, if journal editors (or editorial policies) do not require methods sections in literature reviews, then these sections may not appear. The prerogative of the author(s) seems the most likely determinant of whether a systematic approach is used or not in nutrition and cancer reviews.