Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses – the METSA systematic review
Authors:
- Christian Gunge RiberholtCRChristian Gunge Riberholt
- Markus Harboe OlsenMOMarkus Harboe Olsen
- Joachim Birch MilanJMJoachim Birch Milan
- Sigurlaug Hanna HafliðadóttirSHSigurlaug Hanna Hafliðadóttir
- Jeppe Houmann SvanholmJSJeppe Houmann Svanholm
- Elisabeth Buck PedersenEPElisabeth Buck Pedersen
- Charles Chin Han LewCLCharles Chin Han Lew
- Mark Aninakwah AsanteMAMark Aninakwah Asante
- Johanne Pereira RibeiroJRJohanne Pereira Ribeiro
- Vibeke WagnerVWVibeke Wagner
- Buddheera W. M. B. KumburegamaBKBuddheera W. M. B. Kumburegama
- Zheng-Yii LeeZLZheng-Yii Lee
Open Access
- 9 September 2024
- journal article
- review article
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 24 (1) , 1-17
Abstract
Background: Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis. Methods: We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision. Results: We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%). Conclusions: The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity. Study registration: PROSPERO CRD42021273811Keywords
All Related Versions
Funding Information
- Copenhagen University
This publication has 56 references indexed in Scilit:
- The Impact of Study Size on Meta-analyses: Examination of Underpowered Studies in Cochrane ReviewsPLOS ONE, 2013
- Empirical Evaluation of Very Large Treatment Effects of Medical InterventionsJAMA, 2012
- Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologiesHuman Reproduction, 2012
- GRADE guidelines 6. Rating the quality of evidence—imprecisionJournal of Clinical Epidemiology, 2011
- CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trialsBMJ, 2010
- Apparently conclusive meta-analyses may be inconclusive—Trial sequential analysis adjustment of random error risk due to repetitive testing of accumulating data in apparently conclusive neonatal meta-analysesInternational Journal of Epidemiology, 2008
- Trial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analysesJournal of Clinical Epidemiology, 2008
- Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysisJournal of Clinical Epidemiology, 2008
- Cochrane Skin Group systematic reviews are more methodologically rigorous than other systematic reviews in dermatologyBritish Journal of Dermatology, 2006
- Testosterone and alcoholic cirrhosis. Epidemiologic, pathophysiologic and therapeutic studies in men.1988