Efficacy of Large Language Models for Screening of Systematic Reviews on Periprosthetic Joint Infection
Open Access
- 8 April 2026
- journal article
- research article
- Published by MDPI AG in Journal of Clinical Medicine
- Vol. 15 (8) , 2830
Abstract
Background: Periprosthetic joint infection (PJI) remains a devastating complication following arthroplasty. Systematic reviews of PJI provide essential evidence to inform clinical practice; however, the screening process remains labor-intensive. Recent advancements in large language models (LLMs) offer potential for automating literature screening, though evaluation of current generation models is needed. Methods: This validation study evaluated GPT-5, GPT-5 Pro, and Gemini 2.5 Pro in replicating the title/abstract and full-text screening stages of a published systematic review on intraosseous versus intravenous antibiotic prophylaxis in total joint arthroplasty. Title/abstract screening was performed on 165 articles, followed by a full-text eligibility assessment of 26 articles. Accuracy, sensitivity, specificity, and Cohen’s kappa (κ) were calculated against human screening decisions as the gold standard. Results: In title/abstract screening, GPT-5 Pro achieved the highest accuracy (92.1–92.7%) and specificity (98.6–99.3%), while GPT-5 demonstrated the highest sensitivity (84.6–96.1%). In full-text screening, Gemini 2.5 Pro showed the most consistent performance across repeated evaluations (κ = 0.839 in both trials), whereas GPT-5 Pro exhibited marked intra-model variability (κ = 0.399 to 0.920). Conclusions: Current-generation LLMs achieve near-human accuracy in systematic review screening for PJI research, though substantial intra-model variability underscores the continued need for human oversight in systematic review workflows.Keywords
Funding Information
- National Research Foundation of Korea (RS-2023-00217471)
- Chosun University Hospital
This publication has 30 references indexed in Scilit:
- Can large language models replace humans in systematic reviews? Evaluating GPT‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languagesResearch Synthesis Methods, 2024
- Higher cefazolin concentrations in synovial fluid with intraosseous regional prophylaxis in knee arthroplasty: a randomized controlled trialArchives of orthopaedic and trauma surgery, 2023
- Higher Tissue Concentrations of Vancomycin Achieved With Low-Dose Intraosseous Injection Versus Intravenous Despite Limited Tourniquet Duration in Primary Total Knee Arthroplasty: A Randomized TrialThe Journal of Arthroplasty, 2022
- Intraosseous Regional Administration of Vancomycin in Primary Total Knee Arthroplasty Does Not Increase the Risk of Vancomycin-Associated ComplicationsThe Journal of Arthroplasty, 2021
- Clinical outcome evaluation of intraosseous vancomycin in total knee arthroplastyArthroplasty Today, 2020
- The AAHKS Clinical Research Award: Intraosseous Regional Prophylaxis Provides Higher Tissue Concentrations in High BMI Patients in Total Knee Arthroplasty: A Randomized TrialThe Journal of Arthroplasty, 2018
- The John N. Insall Award: Higher Tissue Concentrations of Vancomycin Achieved With Intraosseous Regional Prophylaxis in Revision TKA: A Randomized Controlled TrialClinical Orthopaedics and Related Research, 2018
- The Mark Coventry Award: Higher Tissue Concentrations of Vancomycin With Low-dose Intraosseous Regional Versus Systemic Prophylaxis in TKAClinical Orthopaedics and Related Research, 2013
- Higher Cefazolin Concentrations with Intraosseous Regional Prophylaxis in TKAClinical Orthopaedics and Related Research, 2013
- The Epidemiology of Revision Total Knee Arthroplasty in the United StatesClinical Orthopaedics and Related Research, 2010