Tools for assessing quality of studies investigating health interventions using real-world data: a literature review and content analysis
Open Access
Abstract
Objectives: We aimed to identify existing appraisal tools for non-randomised studies of interventions (NRSIs) and to compare the criteria that the tools provide at the quality-item level.Design: Literature review through three approaches: systematic search of journal articles, snowballing search of reviews on appraisal tools and grey literature search on websites of health technology assessment (HTA) agencies.Data sources: Systematic search: Medline; Snowballing: starting from three articles (D’Andreaet al, Quigleyet aland Fariaet al); Grey literature: websites of European HTA agencies listed by the International Network of Agencies for Health Technology Assessment. Appraisal tools were searched through April 2022.Eligibility criteria for selecting studies: We included a tool, if it addressed quality concerns of NRSIs and was published in English (unless from grey literature). A tool was excluded, if it was only for diagnostic, prognostic, qualitative or secondary studies.Data extraction and synthesis: Two independent researchers searched, screened and reviewed all included studies and tools, summarised quality items and scored whether and to what extent a quality item was described by a tool, for either methodological quality or reporting.Results: Forty-nine tools met inclusion criteria and were included for the content analysis. Concerns regarding the quality of NRSI were categorised into 4 domains and 26 items. The Research Triangle Institute Item Bank (RTI Item Bank) and STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) were the most comprehensive tools for methodological quality and reporting, respectively, as they addressed (n=20; 17) and sufficiently described (n=18; 13) the highest number of items. However, none of the tools covered all items.Conclusion: Most of the tools have their own strengths, but none of them could address all quality concerns relevant to NRSIs. Even the most comprehensive tools can be complemented by several items. We suggest decision-makers, researchers and tool developers consider the quality-item level heterogeneity, when selecting a tool or identifying a research gap.OSF registration number: OSF registration DOI (https://doi.org/10.17605/OSF.IO/KCSGX).Keywords
Funding Information
- European Union (825162)
This publication has 54 references indexed in Scilit:
- The Current Landscape and Emerging Applications for Real‐World Data in Diagnostics and Clinical Decision Support and its Impact on Regulatory Decision MakingClinical Pharmacology & Therapeutics, 2022
- PROTOCOL: Water, sanitation and hygiene for reducing childhood mortality in low‐ and middle‐income countriesCampbell Systematic Reviews, 2021
- Evaluating the Use of Nonrandomized Real‐World Data Analyses for Regulatory Decision MakingClinical Pharmacology & Therapeutics, 2019
- Using Real-World Data in Health Technology Assessment (HTA) Practice: A Comparative Study of Five HTA AgenciesPharmacoEconomics, 2017
- Academy of Nutrition and Dietetics Methodology for Conducting Systematic Reviews for the Evidence Analysis LibraryJournal of the Academy of Nutrition and Dietetics, 2016
- Guidelines for good pharmacoepidemiology practice (GPP)Pharmacoepidemiology and Drug Safety, 2015
- Non‐randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventionsResearch Synthesis Methods, 2013
- Development of the RTI item bank on risk of bias and precision of observational studiesJournal of Clinical Epidemiology, 2012
- GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)Journal of Clinical Epidemiology, 2011
- A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies ReviewsInternational Journal of Nursing Studies, 2009