Daniele Fanelli

Background: Traditional, complementary, and integrative medicine (TCIM) includes a diverse range of practices, products, and therapies outside the scope of conven-tional Western medicine, including acupuncture, Ayurveda, chiropractic care, and homeopathy. Globally, TCIM is widely used and recognized by most World Health Organization member states, and research in this field has grown substantially in recent decades. However, TCIM research is often criticized for methodological weak-nesses and insufficient rigour, raising concerns about its evidence base and credi-bility. Retraction of scientific articles serves as an essential mechanism to safeguard research integrity, alerting the scientific community to serious errors or misconduct such as fabrication, falsification, or plagiarism. The increasing number of retractions across biomedical research highlights the importance of understanding their under-lying causes. Studying retracted TCIM publications may provide unique insights into recurring methodological, ethical, or systemic issues specific to this field and inform strategies to strengthen research quality. Objectives: The proposed research protocol will use a retrospective analysis to ana-lyse retracted TCIM articles reported in the Retraction Watch database. Methods: Articles classified under ‘(HSC) Medicine – Alternative’ will be identified, extracted, and compared descriptively against retracted non-TCIM publications. Variables of interest include reasons for retraction, date of retraction, article type, journal, publisher, country, number of authors, paywall status, and time from publi-cation to retraction. Data will be summarized using descriptive statistics, with explor-atory comparisons made between TCIM and non-TCIM retractions. Conclusions: The findings will provide the first comprehensive overview of TCIM retractions, offering valuable insights into patterns of flawed research in the field.

Questionable Research Practices (QRPs) are a highly diverse set of behaviours that eludes clear definition and demarcation. This study collects and organises definitions of QRPs given in surveys, and ranks them by frequency of reported engagement. We systematically retrieved surveys that asked researchers about their engagement with QRPs, and organised these survey definitions in two non-arbitrary classifications based, respectively, on the area of research affected by the QRP (e.g. authorship, data, analysis, etc…), and on the nature of the alteration of information entailed by the QRP (respectively, whether the QRP consisted in an omission, addition, or modification of information). Starting from this principled classification, we then created a list of non-overlapping QRP types that have been most commonly studied in surveys. We found that QRPs are more commonly reported when they pertain to the interpretation, analysis or citations, and when they involved the omission, rather than the addition or modification of information. Classified by type, the most commonly reported QRPs included “select analysis”, “select citation”, and “select covariates”; whilst the least commonly reported included “deny authorship”, and “FFP” (i.e. explicit fabrication, falsification, plagiarism, which are not QRPs but outright misconduct). Our QRP taxonomy and empirical results may find useful applications in research, training and policy-making.

Forecasting the reproducibility of research findings is one of the key challenges of metascience. Above-chance predictions have mainly been achieved by pooling the subjective ratings of experts, and how these predictions are formed remains to be understood. Here, we show that reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), a large-scale replication of experiments in the life sciences, are significantly correlated with K , a principled metric of knowledge as information compression. For each study in the BRI sample, we calculated K by dividing the effect size, measured in bits of Shannon entropy, by the descriptive length (a proxy of the complexity) of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol. We found that experts’ predictions about reproducibility were statistically associated with K values and with the complexity of protocols. This relation was robust to controlling for study methodology and other possible confounding factors. These results suggest that expert raters partially judge the reproducibility of findings by assessing the ratio between the information yielded and the information required by a study, and they support the hypothesis that scientific knowledge may be understood and studied through the lenses of information compression.

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta‐analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta‐analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta‐analyses in environmental sciences and psychology, whereas meta‐analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Objectives This descriptive study of registered trials aimed to identify large clinical trials on antidepressants for mental disorders: (1) to assess what proportion could be labelled as ‘seeding trials’ (trials for marketing purposes) and (2) to describe their methodological characteristics and outcomes. Design A search was conducted across all trials registered on ClinicalTrials.gov by drug name in March 2017. Setting All trials registered in the database of ClinicalTrials.gov were screened. Large registered studies were received and studies focusing prospectively on the effects of antidepressants in mental health disorders. Specific data items were extracted automatically, and subsequently inspected, corrected and completed by hand. Participants Prospective studies were selected focusing on the effects of antidepressants in any mental health disorder with 800 participants or more planned for inclusion. Main outcome measures Three members from the study team independently assessed the following ‘seeding trial’ characteristics in each registered study: a high level of involvement of the product manufacturer in the study design, in the data analysis and reporting of the study, an abnormally low ratio of patient numbers to study site, spin and/or omissions of clinically relevant findings in the abstracts, and conclusions that focused on secondary endpoints and surrogate markers. Secondary outcomes were the exploration of a functional outcome and suicidality. Results 31 trials were identified from clinical trials database. 18/31 were published (58%). 8 of these 18 (44%) studies were identified as possible seeding trials. 13/31 (42%) large trials planned to explore functioning and 5/31 (16%) suicidality. Conclusions Large trials are rare in the field of antidepressant research. Some could be ‘seeding trials’. Few explored suicidality. Identifying seeding trials from incomplete data entries in registries, especially when almost half of the studies were still unpublished, posed considerable challenges. The delay between our research and publication limits the strength of our conclusions. PROSPERO registration number CRD42017065591.

The incorporation of publications that have been retracted is a risk in reliable evidence synthesis. Retraction is an important mechanism for correcting the literature and protecting its integrity. Within the medical literature, the continued citation of retracted publications occurs for a variety of reasons. Recent evidence suggests that systematic reviews and meta-analyses often unwittingly cite retracted publications which, at least in some cases, may significantly impact quantitative effect estimates in meta-analyses. There is strong evidence that authors of systematic reviews and meta-analyses may be unaware of the retracted status of publications and treat them as if they are not retracted. These problems are difficult to address for several reasons: identifying retracted publications is important but logistically challenging; publications may be retracted while a review is in preparation or in press and problems with a publication may also be discovered after the evidence synthesis is published. We propose a set of concrete actions that stakeholders (eg, scientists, peer-reviewers, journal editors) might take in the near-term, and that research funders, citation management systems, and databases and search engines might take in the longer term to limit the impact of retracted primary studies on evidence syntheses.

The incorporation of publications that have been retracted is a risk in reliable evidence synthesis. Retraction is an important mechanism for correcting the literature and protecting its integrity. Within the medical literature, the continued citation of retracted publications occurs for a variety of reasons. Recent evidence suggests that systematic reviews and meta-analyses often unwittingly cite retracted publications which, at least in some cases, may significantly impact quantitative effect estimates in meta-analyses. There is strong evidence that authors of systematic reviews and meta-analyses may be unaware of the retracted status of publications and treat them as if they are not retracted. These problems are difficult to address for several reasons: identifying retracted publications is important but logistically challenging; publications may be retracted while a review is in preparation or in press; and problems with a publication may also be discovered after the evidence synthesis is published. We propose a set of concrete actions that stakeholders (e.g., scientists, peer-reviewers, journal editors) might take in the near-term, and that research funders, citation management systems, and databases and search engines might take in the longer term to limit the impact of retracted primary studies on evidence syntheses.

Forecasting the reproducibility of research findings is one of the key challenges of Metascience. To date, reliable above-chance predictions have mainly been achieved by pooling the subjective ratings of experts through surveys or via prediction markets. Obtaining such data is laborious, however, and unlikely to increase the theoretical understanding of reproducibility.Here we show that empirical measures of K, a principled metric of knowledge, are correlated with reproducibility forecasts made for the Brazilian Reproducibility Initiative (BRI), an ongoing replication of 60 studies that use three common methodologies in the life sciences. For each study, we calculated the K value by dividing the effect size, measured in bits of Shannon entropy explained, by the descriptive complexity of the study’s methodology, calculated as the optimal Shannon encoding of a conceptual graph representing the replication protocol.K values were statistically associated with subjective predictions about studies’ replication probabilities and relative effect sizes. This relation was robust to controlling for study methodology, and was a stronger predictor than other plausible covariates, including the originally reported effect size, subjective ratings of a study’s methodological difficulty, and raters’ self-reported expertise. The superior fit of K may be due to its particular structure and metrics, which are rooted in information theory and may more accurately reflect the implicit calculations made by raters.This is the first evidence that an objective metric hypothesized to quantify scientific knowledge captures subjective judgments about reproducibility. This finding gives the first independent support of K’s underlying assumption that scientific knowledge is a process of information compression, and it may offer a new tool for metaresearch.

Pressures to publish, perverse incentives, financial interest and gender are amongst the most commonly discussed risk factors for scientific misconduct. However, evidence of their association with actual data fabrication and falsification is inconclusive. A recent case-controlled analysis of articles containing problematic image duplications suggested that country of affiliation of first and last authors is a significant predictor of scientific misconduct. The same analysis found null or negative associations with individual proxies of publication rate, impact and gender. The latter findings, in line with previous evidence, failed to support common hypotheses about the prevalence and causes of misconduct, but country-level effects may have confounded these results. Here we extend and complete previous results by comparing, via matched-controls analysis, articles from authors in the same country. We found that evidence for individual-level risk factors may be significant in some countries, and null or opposite in others. In particular, in countries where publications are rewarded with cash incentives, and especially China, the risk of problematic image duplication was higher for more productive, more frequently cited, earlier-career researchers working in lower-ranking institutions, in accordance with a “misaligned incentives” explanation for scientific misconduct. However, a null or opposite pattern was observed in all other countries, and especially the USA, UK and Canada, countries where concerns for misaligned incentives are commonly expressed. In line with previous results, we failed to observe a statistically significant association with industry funding and with gender. This is the first direct evidence of a link between publication performance and risk of misconduct and between university ranking and risk of misconduct. Commonly hypothesised individual risk factors for scientific misconduct, including career status and productivity, might be relevant in countries where cash-reward policies generate perverse incentives. In most scientifically active countries, however, where other incentives systems are in place, these patterns are not observed, and other risk factors might be more relevant. Policies to prevent and correct scientific misconduct may need to be tailored to a countries’ or institutions’ specific context.