A case study of the informative value of risk of bias and reporting quality assessments for systematic reviews
Open Access
- 7 September 2024
- journal article
- research article
- Published by Springer Nature in Systematic Reviews
- Vol. 13 (1) , 1-12
Abstract
While undisputedly important, and part of any systematic review (SR) by definition, evaluation of the risk of bias within the included studies is one of the most time-consuming parts of performing an SR. In this paper, we describe a case study comprising an extensive analysis of risk of bias (RoB) and reporting quality (RQ) assessment from a previously published review (CRD42021236047). It included both animal and human studies, and the included studies compared baseline diseased subjects with controls, assessed the effects of investigational treatments, or both. We compared RoB and RQ between the different types of included primary studies. We also assessed the “informative value” of each of the separate elements for meta-researchers, based on the notion that variation in reporting may be more interesting for the meta-researcher than consistently high/low or reported/non-reported scores. In general, reporting of experimental details was low. This resulted in frequent unclear risk-of-bias scores. We observed this both for animal and for human studies and both for disease-control comparisons and investigations of experimental treatments. Plots and explorative chi-square tests showed that reporting was slightly better for human studies of investigational treatments than for the other study types. With the evidence reported as is, risk-of-bias assessments for systematic reviews have low informative value other than repeatedly showing that reporting of experimental details needs to improve in all kinds of in vivo research. Particularly for reviews that do not directly inform treatment decisions, it could be efficient to perform a thorough but partial assessment of the quality of the included studies, either of a random subset of the included publications or of a subset of relatively informative elements, comprising, e.g. ethics evaluation, conflicts of interest statements, study limitations, baseline characteristics, and the unit of analysis. This publication suggests several potential procedures.Keywords
Funding Information
- Medizinische Hochschule Hannover (MHH)
- Medizinische Hochschule Hannover (MHH)
This publication has 36 references indexed in Scilit:
- A systematic review of animal and human data comparing the nasal potential difference test between cystic fibrosis and controlScientific Reports, 2024
- Fallacies of Evidence-Based Medicine in Cardiovascular MedicineThe American Journal of Cardiology, 2019
- The use of Latin-square designs in educational and psychological researchEducational Research Review, 2018
- Research Techniques Made Simple: Assessing Risk of Bias in Systematic ReviewsJournal of Investigative Dermatology, 2016
- ggplot22016
- Ataluren for the treatment of nonsense-mutation cystic fibrosis: a randomised, double-blind, placebo-controlled phase 3 trialThe Lancet Respiratory Medicine, 2014
- Counterbalancing and Other Uses of Repeated-Measures Latin-Square Designs: Analyses and InterpretationsJournal of Experimental Child Psychology, 1997
- Liposome-mediated CFTR gene transfer to the nasal epithelium of patients with cystic fibrosisNature Medicine, 1995
- Non–invasive liposome–mediated gene delivery can correct the ion transport defect in cystic fibrosis mutant miceNature Genetics, 1993
- Cystic fibrosis in the mouse by targeted insertional mutagenesisNature, 1992