Kevin Naaman
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Kevin Naaman has a total of 41 co-authors. Here are the top co-authors by number of shared publications:
- Evan Mayo-Wilson
Department of Epidemiology, University of North Carolina Gillings School of Global Public Health, Chapel Hill, NC, USA
6 shared publications Aaron Beckwith
4 shared publications- Ashma Krishan
Centre for Biostatistics, University of Manchester, Manchester, UK
4 shared publications - Beate Henschel
Indiana University School of Public Health-Bloomington , Bloomington, IN, 47405
4 shared publications - Daniel Shaughnessy
Anschutz Medical Campus University of Colorado Aurora Colorado USA
4 shared publications - Riaz Qureshi
University of Colorado Anschutz Medical Campus
4 shared publications
Latest Publications
- Open Access
- 23 September 2026
- Published by Elsevier in Journal of Clinical Epidemiology
Harms were rarely specified or reported in public health randomized trials: a meta-epidemiological study
- 23 September 2026
- Published by Elsevier in Journal of Clinical Epidemiology
Open AccessHarms were rarely specified or reported in public health randomized trials: a meta-epidemiological study
- Open Access
- 4 July 2026
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 198, 112401
Harms Reporting Was Frequently Incomplete or Discordant in Biomedical Randomized Trials Published in 2023: A Meta-epidemiological Study
- 4 July 2026
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 198, 112401
Open AccessHarms Reporting Was Frequently Incomplete or Discordant in Biomedical Randomized Trials Published in 2023: A Meta-epidemiological Study
- Open Access
- 12 March 2026
- Published by Elsevier in SSRN Electronic Journal
Assessment and Reporting of Harms in Randomized Controlled Trials: A Meta-Epidemiological Study on Trials of Public Health Interventions Published in 2023
BackgroundPublic health interventions can produce adverse and unintended consequences (i.e., harms), but it is unclear how consistently randomized controlled trials (RCTs) of public health interventions plan for and report harms. Incomplete or selective harms reporting can undermine interpretation and evidence synthesis.MethodsWe conducted a meta-epidemiological study of 100 RCTs of public health interventions whose primary full-text report was indexed in PubMed in 2023. Using the publication and publicly linked documents (registrations, protocols, supplements), we assessed harms specification, harms reporting, concordance, transparency of reporting criteria, and whether reported harms were grouped by PROGRESS-Plus equity-related factors.ResultsOnly 25% (25/100) specified at least one harm, and 23% (23/100) reported any harms, while 72% (72/100) neither specified nor reported harms. Among 28 trials that mentioned harms, 12 (43%) reported all specified harms, 8 (29%) reported only a subset, and 8 (29%) showed other discordance. No trial specified harms reporting criteria, and none reported harms using identifiable criteria. Trials with a registry listing and those with an identified protocol had higher odds of harms specification compared with trials without a registry or protocol (OR = 4.7 [1.5, 14.3] and OR = 3.9 [1.5, 10.1], respectively); trials with pharmacological interventions also showed higher odds of specification compared with other intervention types (OR = 9.8 [1.9, 49.9]). Similarly, harms reporting was higher in trials with a registry or protocol (OR = 4.0 [1.3, 12.4] and OR = 3.8 [1.4, 10.0], respectively). Although 89% of trial reports considered at least one PROGRESS-Plus factor among participant characteristics, none reported harms stratified by these factors. Compared with a parallel biomedical sample, public health trials specified and reported harms far less often.ConclusionsIn our sample of RCTs of public health interventions, most trials did not specify harms they planned to measure and did not report harms results. To make public health RCTs more useful in evidence syntheses and decision-making, investigators should specify plausible harms upfront in registries and protocols, then report results for all recorded harms consistently across reports and registries. When harms may differ across groups, reported harms should also be grouped by equity-related participant characteristics.
- 12 March 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessAssessment and Reporting of Harms in Randomized Controlled Trials: A Meta-Epidemiological Study on Trials of Public Health Interventions Published in 2023
BackgroundPublic health interventions can produce adverse and unintended consequences (i.e., harms), but it is unclear how consistently randomized controlled trials (RCTs) of public health interventions plan for and report harms. Incomplete or selective harms reporting can undermine interpretation and evidence synthesis.MethodsWe conducted a meta-epidemiological study of 100 RCTs of public health interventions whose primary full-text report was indexed in PubMed in 2023. Using the publication and publicly linked documents (registrations, protocols, supplements), we assessed harms specification, harms reporting, concordance, transparency of reporting criteria, and whether reported harms were grouped by PROGRESS-Plus equity-related factors.ResultsOnly 25% (25/100) specified at least one harm, and 23% (23/100) reported any harms, while 72% (72/100) neither specified nor reported harms. Among 28 trials that mentioned harms, 12 (43%) reported all specified harms, 8 (29%) reported only a subset, and 8 (29%) showed other discordance. No trial specified harms reporting criteria, and none reported harms using identifiable criteria. Trials with a registry listing and those with an identified protocol had higher odds of harms specification compared with trials without a registry or protocol (OR = 4.7 [1.5, 14.3] and OR = 3.9 [1.5, 10.1], respectively); trials with pharmacological interventions also showed higher odds of specification compared with other intervention types (OR = 9.8 [1.9, 49.9]). Similarly, harms reporting was higher in trials with a registry or protocol (OR = 4.0 [1.3, 12.4] and OR = 3.8 [1.4, 10.0], respectively). Although 89% of trial reports considered at least one PROGRESS-Plus factor among participant characteristics, none reported harms stratified by these factors. Compared with a parallel biomedical sample, public health trials specified and reported harms far less often.ConclusionsIn our sample of RCTs of public health interventions, most trials did not specify harms they planned to measure and did not report harms results. To make public health RCTs more useful in evidence syntheses and decision-making, investigators should specify plausible harms upfront in registries and protocols, then report results for all recorded harms consistently across reports and registries. When harms may differ across groups, reported harms should also be grouped by equity-related participant characteristics.
- Open Access
- 12 March 2026
- Published by Elsevier in SSRN Electronic Journal
Assessment and Reporting of Harms in Randomized Controlled Trials: A Meta-Epidemiological Study on Trials of Biomedical Interventions Published in 2023
BackgroundDecisions about biomedical interventions require clear evidence on benefits and harms, yet harms are often evaluated and reported less rigorously than efficacy outcomes. Unstated reporting rules for harms may lead to "cherry-picked" results in primary studies.MethodsWe conducted a meta-epidemiological study of a random sample of 200 randomized controlled trials (RCTs) of biomedical interventions indexed in PubMed in 2023. Using publicly available protocols, trial registries, and publications, we assessed harms specification, harms reporting, concordance between specified and reported harms, transparency about harms reporting criteria, and whether trials had considered PROGRESS-Plus equity-related factors (an equity framework for participant characteristics linked to health inequities) in reporting harms.ResultsSixty-four percent (128/200) of trials were registered and 15.5% had an accessible protocol (31/200). Harms were specified in 64.0% (128/200) of primary publications and reported in 69.5% (139/200). Concordance was often incomplete: 43.5% (87/200) reported all specified harms, and 26.5% (53/200) specified no harms and reported none, while 14.5% (29/200) reported only some specified harms, and 15.5% (31/200) showed other discordance. Only 1.0% (2/200) clearly specified harms reporting criteria, and 7.0% (14/200) indicated using criteria when reporting. Trials with a registry listing or an identified protocol had higher odds of specifying harms compared with trials without a registry listing or an identified protocol (OR = 2.1 [1.2, 3.8] and OR = 5.6 [1.8, 18.1], respectively). Compared with behavioral or physical therapy interventions, pharmaceutical, surgical, or device trials had higher odds of harms specification (OR = 5.3 [2.8, 10.1]) and harms reporting (OR = 6.8 [3.5, 13.2]). No trials reported harms stratified by PROGRESS-Plus equity-related factors.ConclusionsIn a contemporary sample of biomedical RCTs, harms were frequently specified and reported, but concordance between planned and reported harms and transparency about reporting criteria remain deficient. Trialists should pre-specify harms outcomes in protocols/registrations and report harms comprehensively and consistently across all public sources; descriptive harms summaries by key equity-related factors should be used to help identify potential vulnerability. Journal editors and peer reviewers should require clearer harms methods and complete harms results in RCT reports.
- 12 March 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessAssessment and Reporting of Harms in Randomized Controlled Trials: A Meta-Epidemiological Study on Trials of Biomedical Interventions Published in 2023
BackgroundDecisions about biomedical interventions require clear evidence on benefits and harms, yet harms are often evaluated and reported less rigorously than efficacy outcomes. Unstated reporting rules for harms may lead to "cherry-picked" results in primary studies.MethodsWe conducted a meta-epidemiological study of a random sample of 200 randomized controlled trials (RCTs) of biomedical interventions indexed in PubMed in 2023. Using publicly available protocols, trial registries, and publications, we assessed harms specification, harms reporting, concordance between specified and reported harms, transparency about harms reporting criteria, and whether trials had considered PROGRESS-Plus equity-related factors (an equity framework for participant characteristics linked to health inequities) in reporting harms.ResultsSixty-four percent (128/200) of trials were registered and 15.5% had an accessible protocol (31/200). Harms were specified in 64.0% (128/200) of primary publications and reported in 69.5% (139/200). Concordance was often incomplete: 43.5% (87/200) reported all specified harms, and 26.5% (53/200) specified no harms and reported none, while 14.5% (29/200) reported only some specified harms, and 15.5% (31/200) showed other discordance. Only 1.0% (2/200) clearly specified harms reporting criteria, and 7.0% (14/200) indicated using criteria when reporting. Trials with a registry listing or an identified protocol had higher odds of specifying harms compared with trials without a registry listing or an identified protocol (OR = 2.1 [1.2, 3.8] and OR = 5.6 [1.8, 18.1], respectively). Compared with behavioral or physical therapy interventions, pharmaceutical, surgical, or device trials had higher odds of harms specification (OR = 5.3 [2.8, 10.1]) and harms reporting (OR = 6.8 [3.5, 13.2]). No trials reported harms stratified by PROGRESS-Plus equity-related factors.ConclusionsIn a contemporary sample of biomedical RCTs, harms were frequently specified and reported, but concordance between planned and reported harms and transparency about reporting criteria remain deficient. Trialists should pre-specify harms outcomes in protocols/registrations and report harms comprehensively and consistently across all public sources; descriptive harms summaries by key equity-related factors should be used to help identify potential vulnerability. Journal editors and peer reviewers should require clearer harms methods and complete harms results in RCT reports.
- Open Access
- 12 February 2026
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e69707
Using Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and Human-Annotated Bibliographies
Background: Annotated bibliographies summarize literature, but training, experience, and time are needed to create concise yet accurate annotations. Summaries generated by artificial intelligence (AI) can save human resources, but AI-generated content can also contain serious errors. Objective: To determine the feasibility of using AI as an alternative to human annotators, we explored whether ChatGPT can generate annotations with characteristics that are comparable to those written by humans. Methods: We had 2 humans and 3 versions of ChatGPT (3.5, 4, and 5) independently write annotations on the same set of 15 publications. We collected data on word count and Flesch Reading Ease (FRE). In this study, 2 assessors who were masked to the source of the annotations independently evaluated (1) capture of main points, (2) presence of errors, and (3) whether the annotation included a discussion of both the quality and context of the article within the broader literature. We evaluated agreement and disagreement between the assessors and used descriptive statistics and assessor-stratified binary and cumulative mixed-effects logit models to compare annotations written by ChatGPT and humans. Results: On average, humans wrote shorter annotations (mean 90.20, SD 36.8 words) than ChatGPT (mean 113, SD 16 words) which were easier to interpret (human FRE score, mean 15.3, SD 12.4; ChatGPT FRE score, mean 5.76, SD 7.32). Our assessments of agreement and disagreement revealed that one assessor was consistently stricter than the other. However, assessor-stratified models of main points, errors, and quality/context showed similar qualitative conclusions. There was no statistically significant difference in the odds of presenting a better summary of main points between ChatGPT- and human-generated annotations for either assessor (Assessor 1: OR 0.96, 95% CI 0.12‐7.71; Assessor 2: OR 1.64, 95% CI 0.67‐4.06). However, both assessors observed that human annotations had lower odds of having one or more types of errors compared to ChatGPT (Assessor 1: OR 0.31, 95% CI 0.09‐1.02; Assessor 2: OR 0.10, 95% CI 0.03‐0.33). On the other hand, human annotations also had lower odds of summarizing the paper’s quality and context when compared to ChatGPT (Assessor 1: OR 0.11, 95% CI 0.03‐0.33; Assessor 2: OR 0.03, 95% CI 0.01‐0.10). That said, ChatGPT’s summaries of quality and context were sometimes inaccurate. Conclusions: Rapidly learning a body of scientific literature is a vital yet daunting task that may be made more efficient by AI tools. In our study, ChatGPT quickly generated concise summaries of academic literature and also provided quality and context more consistently than humans. However, ChatGPT’s discussion of the quality and context was not always accurate, and ChatGPT annotations included more errors. Annotated bibliographies that are AI-generated and carefully verified by humans may thus be an efficient way to provide a rapid overview of literature. More research is needed to determine the extent that prompt engineering can reduce errors and improve chatbot performance.
- 12 February 2026
- Published by JMIR Publications Inc. in JMIR Formative Research
- Vol. 10, e69707
Open AccessUsing Large Language Models to Summarize Evidence in Biomedical Articles: Exploratory Comparison Between AI- and Human-Annotated Bibliographies
Background: Annotated bibliographies summarize literature, but training, experience, and time are needed to create concise yet accurate annotations. Summaries generated by artificial intelligence (AI) can save human resources, but AI-generated content can also contain serious errors. Objective: To determine the feasibility of using AI as an alternative to human annotators, we explored whether ChatGPT can generate annotations with characteristics that are comparable to those written by humans. Methods: We had 2 humans and 3 versions of ChatGPT (3.5, 4, and 5) independently write annotations on the same set of 15 publications. We collected data on word count and Flesch Reading Ease (FRE). In this study, 2 assessors who were masked to the source of the annotations independently evaluated (1) capture of main points, (2) presence of errors, and (3) whether the annotation included a discussion of both the quality and context of the article within the broader literature. We evaluated agreement and disagreement between the assessors and used descriptive statistics and assessor-stratified binary and cumulative mixed-effects logit models to compare annotations written by ChatGPT and humans. Results: On average, humans wrote shorter annotations (mean 90.20, SD 36.8 words) than ChatGPT (mean 113, SD 16 words) which were easier to interpret (human FRE score, mean 15.3, SD 12.4; ChatGPT FRE score, mean 5.76, SD 7.32). Our assessments of agreement and disagreement revealed that one assessor was consistently stricter than the other. However, assessor-stratified models of main points, errors, and quality/context showed similar qualitative conclusions. There was no statistically significant difference in the odds of presenting a better summary of main points between ChatGPT- and human-generated annotations for either assessor (Assessor 1: OR 0.96, 95% CI 0.12‐7.71; Assessor 2: OR 1.64, 95% CI 0.67‐4.06). However, both assessors observed that human annotations had lower odds of having one or more types of errors compared to ChatGPT (Assessor 1: OR 0.31, 95% CI 0.09‐1.02; Assessor 2: OR 0.10, 95% CI 0.03‐0.33). On the other hand, human annotations also had lower odds of summarizing the paper’s quality and context when compared to ChatGPT (Assessor 1: OR 0.11, 95% CI 0.03‐0.33; Assessor 2: OR 0.03, 95% CI 0.01‐0.10). That said, ChatGPT’s summaries of quality and context were sometimes inaccurate. Conclusions: Rapidly learning a body of scientific literature is a vital yet daunting task that may be made more efficient by AI tools. In our study, ChatGPT quickly generated concise summaries of academic literature and also provided quality and context more consistently than humans. However, ChatGPT’s discussion of the quality and context was not always accurate, and ChatGPT annotations included more errors. Annotated bibliographies that are AI-generated and carefully verified by humans may thus be an efficient way to provide a rapid overview of literature. More research is needed to determine the extent that prompt engineering can reduce errors and improve chatbot performance.
- 1 August 2024
- Published by American College of Physicians in Annals of Internal Medicine
- Vol. 177 (8) , 1089-1098
Development and Evaluation of a Framework for Identifying and Addressing Spin for Harms in Systematic Reviews of Interventions
"Spin" refers to misleading reporting, interpretation, and extrapolation of findings in primary and secondary research (such as in systematic reviews). The study of spin primarily focuses on beneficial outcomes. The objectives of this research were threefold: first, to develop a framework for identifying spin associated with harms in systematic reviews of interventions; second, to apply the framework to a set of reviews, thereby pinpointing instances where spin may be present; and finally, to revise the spin examples, offering guidance on how spin can be rectified. The authors developed their framework through an iterative process that engaged an international group of researchers specializing in spin and reporting bias. The framework comprises 12 specific types of spin for harms, grouped by 7 categories across the 3 domains (reporting, interpretation, and extrapolation). The authors subsequently gathered instances of spin from a random sample of 100 systematic reviews of interventions. Of the 58 reviews that assessed harm and the 42 that did not, they found that 28 (48%) and 6 (14%), respectively, had at least 1 of the 12 types of spin for harms. Inappropriate extrapolation of the results and conclusions for harms to populations, interventions, outcomes, or settings not assessed in a review was the most common category of spin in 17 of 100 reviews. The authors revised the examples to remove spin, taking into consideration the context (for example, medical discipline, source population), findings for harms, and methodological limitations of the original reviews. They provide guidance for authors, peer reviewers, and editors in recognizing and rectifying or (preferably) avoiding spin, ultimately enhancing the clarity and accuracy of harms reporting in systematic review publications.
- 1 August 2024
- Published by American College of Physicians in Annals of Internal Medicine
- Vol. 177 (8) , 1089-1098
Development and Evaluation of a Framework for Identifying and Addressing Spin for Harms in Systematic Reviews of Interventions
"Spin" refers to misleading reporting, interpretation, and extrapolation of findings in primary and secondary research (such as in systematic reviews). The study of spin primarily focuses on beneficial outcomes. The objectives of this research were threefold: first, to develop a framework for identifying spin associated with harms in systematic reviews of interventions; second, to apply the framework to a set of reviews, thereby pinpointing instances where spin may be present; and finally, to revise the spin examples, offering guidance on how spin can be rectified. The authors developed their framework through an iterative process that engaged an international group of researchers specializing in spin and reporting bias. The framework comprises 12 specific types of spin for harms, grouped by 7 categories across the 3 domains (reporting, interpretation, and extrapolation). The authors subsequently gathered instances of spin from a random sample of 100 systematic reviews of interventions. Of the 58 reviews that assessed harm and the 42 that did not, they found that 28 (48%) and 6 (14%), respectively, had at least 1 of the 12 types of spin for harms. Inappropriate extrapolation of the results and conclusions for harms to populations, interventions, outcomes, or settings not assessed in a review was the most common category of spin in 17 of 100 reviews. The authors revised the examples to remove spin, taking into consideration the context (for example, medical discipline, source population), findings for harms, and methodological limitations of the original reviews. They provide guidance for authors, peer reviewers, and editors in recognizing and rectifying or (preferably) avoiding spin, ultimately enhancing the clarity and accuracy of harms reporting in systematic review publications.
- Open Access
- 1 February 2023
- Published by The Royal Society in Royal Society Open Science
- Vol. 10 (2) , 221093
Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors
The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.
- 1 February 2023
- Published by The Royal Society in Royal Society Open Science
- Vol. 10 (2) , 221093
Open AccessExploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors
The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.
- Open Access
- 19 August 2022
- Published by Center for Open Science in MetaArXiv
Exploring enablers and barriers to implementing the Transparency and Openness Promotion (TOP) Guidelines: A theory-based survey of journal editors
The Transparency and Openness Promotion (TOP) Guidelines provide a framework for journals to develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity, and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared to other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g., lack of time, resources, and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically-guided interventions to increase open science policies, procedures, and practices.
- 19 August 2022
- Published by Center for Open Science in MetaArXiv
Open AccessExploring enablers and barriers to implementing the Transparency and Openness Promotion (TOP) Guidelines: A theory-based survey of journal editors
The Transparency and Openness Promotion (TOP) Guidelines provide a framework for journals to develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity, and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared to other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g., lack of time, resources, and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically-guided interventions to increase open science policies, procedures, and practices.
- Open Access
- 19 August 2022
- Published by Center for Open Science in MetaArXiv
Evaluating implementation of the Transparency and Openness Promotion Guidelines: Reliability of instruments to assess journal policies, procedures, and practices
The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor quantifies the extent to which journals have adopted the TOP Guidelines in their policies, but there is no validated instrument to assess journal implementation of TOP. Moreover, raters might differ in their assessments of the same journal policies. Instruments with objective and clear questions are needed to assess TOP implementation reliably. This study examined the interrater reliability and agreement of three new instruments for assessing TOP implementation in journal policies (instructions to authors), procedures (manuscript submission systems), and practices (journal articles). Independent raters used these instruments to assess the policies, procedures, and practices of 339 journals from the behavioral, social, and health sciences. We calculated interrater agreement (IRA) and interrater reliability (IRR) for each standard in TOP (10 journal policy standards) and for each question in our instruments (13 policy questions, 26 procedure questions, 14 practice questions). IRA was high for each standard in TOP; however, IRA might have been high by chance because most standards were not implemented by most journals. No standard had “excellent” IRR. Three standards had “good”, one had “moderate”, and six had “poor” IRR. Similarly, IRA was high for most instrument questions, and IRR was “moderate” or worse for 62%, 54%, and 43% of policy, procedure, and practice questions, respectively. We conclude that clarifying the distinctions among different levels of implementation for each TOP standard might improve its implementation and assessment.
- 19 August 2022
- Published by Center for Open Science in MetaArXiv
Open AccessEvaluating implementation of the Transparency and Openness Promotion Guidelines: Reliability of instruments to assess journal policies, procedures, and practices
The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor quantifies the extent to which journals have adopted the TOP Guidelines in their policies, but there is no validated instrument to assess journal implementation of TOP. Moreover, raters might differ in their assessments of the same journal policies. Instruments with objective and clear questions are needed to assess TOP implementation reliably. This study examined the interrater reliability and agreement of three new instruments for assessing TOP implementation in journal policies (instructions to authors), procedures (manuscript submission systems), and practices (journal articles). Independent raters used these instruments to assess the policies, procedures, and practices of 339 journals from the behavioral, social, and health sciences. We calculated interrater agreement (IRA) and interrater reliability (IRR) for each standard in TOP (10 journal policy standards) and for each question in our instruments (13 policy questions, 26 procedure questions, 14 practice questions). IRA was high for each standard in TOP; however, IRA might have been high by chance because most standards were not implemented by most journals. No standard had “excellent” IRR. Three standards had “good”, one had “moderate”, and six had “poor” IRR. Similarly, IRA was high for most instrument questions, and IRR was “moderate” or worse for 62%, 54%, and 43% of policy, procedure, and practice questions, respectively. We conclude that clarifying the distinctions among different levels of implementation for each TOP standard might improve its implementation and assessment.
- Open Access
- 18 August 2022
- Published by Center for Open Science in MetaArXiv
Open Science Standards at Journals that Inform Evidence-Based Policy
Evidence-based policy uses intervention research published in peer-reviewed journals to inform consequential decisions about resource allocation. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science—such as the Transparency and Openness Promotion (TOP) Guidelines—could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals used to identify evidence-based interventions for policymaking purposes. Each of ten open science standards in TOP was not implemented in most journals’ policies (i.e., instructions to authors), procedures (i.e., manuscript submission systems), or practices (i.e., published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve their implementation of open science standards to safeguard evidence-based policy.
- 18 August 2022
- Published by Center for Open Science in MetaArXiv
Open AccessOpen Science Standards at Journals that Inform Evidence-Based Policy
Evidence-based policy uses intervention research published in peer-reviewed journals to inform consequential decisions about resource allocation. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science—such as the Transparency and Openness Promotion (TOP) Guidelines—could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals used to identify evidence-based interventions for policymaking purposes. Each of ten open science standards in TOP was not implemented in most journals’ policies (i.e., instructions to authors), procedures (i.e., manuscript submission systems), or practices (i.e., published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve their implementation of open science standards to safeguard evidence-based policy.