Nico Riedel

Increasing the availability of research data sets is a goal of many stakeholders in science, and monitoring related practices requires definitions of the entity in question. There are several, largely overlapping, definitions for open data. However, they have so far not been translated into operationalizations that would allow us to detect, in a structured and reproducible way, whether, for a specific research article, the underlying data have been shared. Here, we propose a detailed set of criteria to enable such assessments, focusing on biomedical research. We have used these criteria to distribute performance-oriented funding at a large university hospital and to monitor data sharing practices in a dashboard. In addition to fully open data, we include separate criteria for data sets with restricted access, which we also reward. The criteria are partly inspired by the FAIR principles, particularly findability and accessibility, but do not map onto individual principles. The criteria attribute open data status in a binary fashion, both to individual data sets and, ultimately, articles with which they were shared. The criteria allow a verifiable assessment, based on automated and manual screening steps, which we have implemented and validated, as described elsewhere. Here, we focus conceptually on assessing the presence of shared data.

Background: University Medical Centers (UMCs) must do their part for clinical trial transparency by fostering practices such as prospective registration, timely results reporting, and open access. However, research institutions are often unaware of their performance on these practices. Baseline assessments of these practices would highlight where there is room for change and empower UMCs to support improvement. We performed a status quo analysis of established clinical trial registration and reporting practices at German UMCs and developed a dashboard to communicate these baseline assessments with UMC leadership and the wider research community. Methods and findings: We developed and applied a semiautomated approach to assess adherence to established transparency practices in a cohort of interventional trials and associated results publications. Trials were registered in ClinicalTrials.gov or the German Clinical Trials Register (DRKS), led by a German UMC, and reported as complete between 2009 and 2017. To assess adherence to transparency practices, we identified results publications associated to trials and applied automated methods at the level of registry data (e.g., prospective registration) and publications (e.g., open access). We also obtained summary results reporting rates of due trials registered in the EU Clinical Trials Register (EUCTR) and conducted at German UMCs from the EU Trials Tracker. We developed an interactive dashboard to display these results across all UMCs and at the level of single UMCs. Our study included and assessed 2,895 interventional trials led by 35 German UMCs. Across all UMCs, prospective registration increased from 33% (n = 58/178) to 75% (n = 144/193) for trials registered in ClinicalTrials.gov and from 0% (n = 0/44) to 79% (n = 19/24) for trials registered in DRKS over the period considered. Of trials with a results publication, 38% (n = 714/1,895) reported the trial registration number in the publication In turn, 58% (n = 861/1,493) of trials registered in ClinicalTrials.gov and 23% (n = 111/474) of trials registered in DRKS linked the publication in the registration. In contrast to recent increases in summary results reporting of drug trials in the EUCTR, 8% (n = 191/2,253) and 3% (n = 20/642) of due trials registered in ClinicalTrials.gov and DRKS, respectively, had summary results in the registry. Across trial completion years, timely results reporting (within 2 years of trial completion) as a manuscript publication or as summary results was 41% (n = 1,198/2,892). The proportion of openly accessible trial publications steadily increased from 42% (n = 16/38) to 74% (n = 72/97) over the period considered. A limitation of this study is that some of the methods used to assess the transparency practices in this dashboard rely on registry data being accurate and up-to-date. Conclusions: In this study, we observed that it is feasible to assess and inform individual UMCs on their performance on clinical trial transparency in a reproducible and publicly accessible way. Beyond helping institutions assess how they perform in relation to mandates or their institutional policy, the dashboard may inform interventions to increase the uptake of clinical transparency practices and serve to evaluate the impact of these interventions.

There are several definitions for open data, largely in agreement with each other. However, they have so far not been translated into operationalizations which would allow to detect in a structured and reproducible way, whether for a specific research article underlying data have been shared. Here, we propose a detailed set of criteria to enable such an assessment, focusing on biomedical research. We have used this assessment for the distribution of performance-oriented funding at a large university hospital, as well as for monitoring of data sharing practices in a dashboard. In addition to fully open data, we include a separate set of criteria for datasets with restricted access, which we also reward. The criteria are in part inspired by the FAIR principles, with a focus on findability and accessibility, but do not map onto individual principles. The criteria attribute open data status in a binary fashion, both to individual datasets and, ultimately, the article with which they were shared. The criteria allow a verifiable assessment, based on an automated and a manual screening step, which we have implemented and validated, as described elsewhere. In the present article, we focus on the conceptual level of assessing the presence of shared data.

Recent work has raised awareness about the need to replace bar graphs of continuous data with informative graphs showing the data distribution. The impact of these efforts is not known. The present observational meta-research study examined how often scientists in different fields use various graph types, and assessed whether visualization practices have changed between 2010 and 2020. We developed and validated an automated screening tool, designed to identify bar graphs of counts or proportions, bar graphs of continuous data, bar graphs with dot plots, dot plots, box plots, violin plots, histograms, pie charts, and flow charts. Papers from 23 fields (approximately 1000 papers/field per year) were randomly selected from PubMed Central and screened (n=227998). F1 scores for different graphs ranged between 0.83 and 0.95 in the internal validation set. While the tool also performed well in external validation sets, F1 scores were lower for uncommon graphs. Bar graphs are more often used incorrectly to display continuous data than they are used correctly to display counts or proportions. The proportion of papers that use bar graphs of continuous data varies markedly across fields (range in 2020: 4–58%), with high rates in biochemistry and cell biology, complementary and alternative medicine, physiology, genetics, oncology and carcinogenesis, pharmacology, microbiology and immunology. Visualization practices have improved in some fields in recent years. Fewer than 25% of papers use flow charts, which provide information about attrition and the risk of bias. The present study highlights the need for continued interventions to improve visualization and identifies fields that would benefit most.

Background Mandates and recommendations related to embedding open science practices within the research lifecycle are increasingly common. Few stakeholders, however, are monitoring compliance to their mandates or recommendations. It is necessary to monitor the current state of open science to track changes over time and to identify areas to create interventions to drive improvements. Monitoring open science practices requires that they are defined and operationalized. Involving the biomedical community, we sought to reach consensus on a core set of open science practices to monitor at biomedical research institutions. Methods and Findings To establish consensus in a structured and systematic fashion, we conducted a modified 3-round Delphi study. Participants in Round 1 were 80 individuals from 20 biomedical research institutions that exhibit interest in or actively support open science. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In Rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices that could be important and meaningful to monitor in an automated institutional open science dashboard. Participants voted on the inclusion of each item and provided a rationale for their choice. We defined consensus as 80% agreement. Between rounds, participants received aggregated voting scores for each item and anonymized comments from all participants, and were asked to re-vote on items that did not reach consensus. For Round 3, we hosted two half- day virtual meetings with 21 and 17 participants respectively to discuss and vote on all items that had not reached consensus after Round 2. Ultimately, participants reached consensus to include a 19 open science practices. Conclusions A group of international stakeholders used a modified Delphi process to agree upon open science practices to monitor in a proposed open science dashboard for biomedical institutions. The core set of 19 open science practices identified by participants will form the foundation for institutional dashboards that display compliance with open science practices. They will now be assessed and tested for automatic inclusion in terms of technical feasibility. Using user-centered design, participating institutions will be involved in creating a dashboard prototype, which can then be implemented to monitor rates of open science practices at biomedical institutions. Our methods and approach may also transfer to other research settings–other disciplines could consider using our consensus list as a starting point for agreement upon a discipline-specific set of open science practices to monitor. The findings may also be of broader value to the development of policy, education, and interventions.

Western blotting is a standard laboratory method used to detect proteins and assess their expression levels. Unfortunately, poor western blot image display practices and a lack of detailed methods reporting can limit a reader’s ability to evaluate or reproduce western blot results. While several groups have studied the prevalence of image manipulation or provided recommendations for improving western blotting, data on the prevalence of common publication practices are scarce. We systematically examined 551 articles published in the top 25% of journals in neurosciences (n=151) and cell biology (n=400) that contained western blot images, focusing on practices that may omit important information. Our data show that most published western blots are cropped and blot source data are not made available to readers in the supplement. Publishing blots with visible molecular weight markers is rare, and many blots additionally lack molecular weight labels. Western blot methods sections often lack information on the amount of protein loaded on the gel, blocking steps and antibody labeling protocol. Important antibody identifiers like source, catalog number or RRID were omitted frequently for primary antibodies, and regularly for secondary antibodies. We present detailed descriptions and visual examples to help scientists, peer reviewers and editors to publish more informative western blot figures and methods. Additional resources include a toolbox to help scientists produce more reproducible western blot data, teaching slides in English and Spanish and an antibody reporting template.

Recent work has raised awareness about the need to replace bar graphs of continuous data with informative graphs showing the data distribution. The impact of these efforts is not known. This observational meta-research study examined how often scientists in different fields use various graph types, and assessed whether visualization practices have changed between 2010 and 2020. We developed and validated an automated screening tool, designed to identify bar graphs of counts or proportions, bar graphs of continuous data, bar graphs with dot plots, dot plots, box plots, violin plots, histograms, pie charts, and flow charts. Papers from 23 fields (approximately 1,000 papers/field/year) were randomly selected from PubMed Central and screened (n=227,998). F1 scores for different graphs ranged between 0.83 and 0.95 in the internal validation set. While the tool also performed well in external validation sets, F1 scores were lower for uncommon graphs. Bar graphs are more often used incorrectly to display continuous data than they are used correctly to display counts or proportions. The proportion of papers that use bar graphs of continuous data varies markedly across fields (range in 2020: 4%-58%), with high rates in biochemistry and cell biology, complementary and alternative medicine, physiology, genetics, oncology and carcinogenesis, pharmacology, microbiology and immunology. Visualization practices have improved in some fields in recent years. Fewer than 25% of papers use flow charts, which provide information about attrition and the risk of bias. This study highlights the need for continued interventions to improve visualization and identifies fields that would benefit most.

Objective Timely publication of clinical trial results is central for evidence-based medicine. The performance of university medical centers (UMCs) regarding timely results publication informs the local and national decision makers, patients, and the public about the need to improve trial results dissemination. Study Design and Setting Following the same search and tracking methods used in our previous study for the years 2009 - 2013, we identified trials led by German UMCs completed between 2014 and 2017 and tracked results dissemination for the identified trials. Results We identified 1,658 trials in the 2014-2017 cohort. Of these trials, 43% published results as either journal publication or summary results within 24 months after completion date, which is an improvement of 3.8% percentage points compared to the previous study. At the UMC level, the 24-month publication rates varied from 14% to 71%. Five years after completion, still 30% of the trials remained unpublished. Conclusion Despite minor improvements compared to the previously investigated cohort, the rate of timely reporting of trials led by German UMCs remains low. German UMCs should take further steps to improve timely reporting rates.

Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.