Analysis of indications for selectively missing results in comparative registry-based studies in medicine: a meta-research study
Authors:
Open Access
- 5 March 2025
- journal article
- research article
- Published by Springer Nature in Research Integrity and Peer Review
- Vol. 10 (1) , 1-7
Abstract
Background: We assess if there are indications that results of registry-based studies comparing the effectiveness of interventions might be selectively missing depending on the statistical significance (p < 0.05). Methods: Eligibility criteria Sample of cohort type studies that used data from a patient registry, compared two study arms for assessing a medical intervention, and reported an effect for a binary outcome. Information sources We searched PubMed to identify registries in seven different medical specialties in 2022/23. Subsequently, we included all studies that satisfied the eligibility criteria for each of the identified registries and collected p-values from these studies. Synthesis of results We plotted the cumulative distribution of p-values and a histogram of absolute z-scores for visual inspection of selectively missing results because of p-hacking, selective reporting, or publication bias. In addition, we tested for publication bias by applying a caliper test. Results: Included studies Sample of 150 registry-based cohort type studies. Synthesis of results The cumulative distribution of p-values displays an abrupt, heavy increase just below the significance threshold of 0.05 while the distribution above the threshold shows a slow, gradual increase. The p-value of the caliper test with a 10% caliper was 0.011 (k = 2, N = 13). Conclusions: We found that the results of registry-based studies might be selectively missing. Results from registry-based studies comparing medical interventions should be interpreted very cautiously, as positive findings could be a result from p-hacking, publication bias, or selective reporting. Prospective registration of such studies is necessary and should be made mandatory both in regulatory contexts and for publication in journals. Further research is needed to determine the main reasons for selectively missing results to support the development and implementation of more specific methods for preventing selectively missing results.Keywords
Funding Information
- Herzzentrum Göttingen
This publication has 23 references indexed in Scilit:
- Systematic reviews and meta-analyses that include registry-based studies: methodological challenges and areas for future researchJournal of Clinical Epidemiology, 2023
- Big little lies: a compendium and simulation ofp-hacking strategiesRoyal Society Open Science, 2023
- HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real‐world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task forcePharmacoepidemiology and Drug Safety, 2022
- The rise of registry-based research: a bibliometric analysisActa Orthopaedica, 2021
- Development and validation of study filters for identifying controlled non‐randomized studies in PubMed and Ovid MEDLINEResearch Synthesis Methods, 2020
- QUALITY ASSURANCE OF REGISTRIES FOR HEALTH TECHNOLOGY ASSESSMENTInternational Journal of Technology Assessment in Health Care, 2018
- Registration practices for observational studies on ClinicalTrials.gov indicated low adherenceJournal of Clinical Epidemiology, 2016
- P-curve: A key to the file-drawer.Journal of Experimental Psychology: General, 2014
- Empirical Evidence for Selective Reporting of Outcomes in Randomized TrialsJAMA, 2004
- Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐AnalysisBiometrics, 2000