Assessing the risk of bias of clinical trials with large language models and ROBUST-RCT: a feasibility study
Open Access
- 17 March 2026
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 16 (1)
Abstract
Risk of bias assessment is a crucial step in evidence synthesis. The traditionally adopted tool, however, is complex, resource-intensive, and unreliable. While prior investigations have focused on whether Large Language Models (LLMs) could perform assessments with RoB 2, this study is the first to evaluate the reliability of ROBUST-RCT, a novel risk-of-bias tool, as applied by humans and LLMs. Reviewers working independently used ROBUST-RCT to assess different aspects of a sample of RCTs and then reached a consensus through discussion. A chain-of-thought prompt instructed four LLMs on how to apply ROBUST-RCT. The primary analysis used Gwet’s AC2 to assess inter-rater reliability based on all the final ratings (i.e., the ratings in the second step of the tool) for all the core items of the ROBUST-RCT. A sample of 54 assessments, derived from 9 studies, was compared for each LLM against human consensus. In the primary analysis, Gwet’s AC2 inter-rater reliability varied across the LLMs. DeepSeek-R1, the lowest performer, yielded an AC2 of 0.46 ( 95% CI: 0.24 to 0.69). On the other side, Gemini 2.5 Pro Preview – the model with higher consistency with human consensus – yielded an AC2 of 0.69 (95% CI: 0.54 to 0.84). With 95% confidence, three of the four tested LLMs achieved ‘moderate’ or higher reliability based on benchmarking. LLMs could be helpful in the risk-of-bias assessment of systematic reviews using the ROBUST-RCT tool.Keywords
Funding Information
- Fundação Médica do Rio Grande do Sul
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
This publication has 51 references indexed in Scilit:
- Artificial intelligence in scientific medical writing: Legitimate and deceptive uses and ethical concernsEuropean Journal of Internal Medicine, 2024
- The Cochrane risk of bias assessment tool 2 (RoB 2) versus the original RoB: A perspective on the pros and consHealth Science Reports, 2024
- Frequency of use and adequacy of Cochrane risk of bias tool 2 in non‐Cochrane systematic reviews published in 2020: Meta‐research studyResearch Synthesis Methods, 2024
- A review of top cardiology and cardiovascular medicine journal guidelines regarding the use of generative artificial intelligence tools in scientific writingCurrent Problems in Cardiology, 2024
- Robustness of κ‐type coefficients for clinical agreementStatistics in Medicine, 2022
- Interrater reliability of ROB2 – an alternative measure and way of categorizationJournal of Clinical Epidemiology, 2022
- The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its applicationJournal of Clinical Epidemiology, 2020
- Half of Cochrane reviews were published more than 2 years after the protocolJournal of Clinical Epidemiology, 2020
- High agreement but low kappa: II. Resolving the paradoxesJournal of Clinical Epidemiology, 1990
- High agreement but low Kappa: I. the problems of two paradoxesJournal of Clinical Epidemiology, 1990