Tomer Bernstine

BACKGROUND Systematic reviews (SRs) are a resource-intensive cornerstone of evidence-based medicine and public health. Their structured, rule-based workflow makes them amenable to large language models (LLMs) streamlining the process. Most evaluations benchmark a single model, task or corpus, leaving unresolved whether performance is valid and consistent across models, steps, review topics and workflow designs. OBJECTIVE To evaluate the validity and consistency of ChatGPT and Gemini across successive stages of SR screening, in order to determine whether LLMs can be safely integrated into SR workflows. METHODS We evaluated ChatGPT-4 and Gemini 3 against three public health SRs conducted by our research group. Human decisions served as the reference standard. Title, abstract and full-text screening were compared under two workflows: teacher forcing (TF), in which the model was constrained to the human-retained denominator at each stage, and non-teacher forcing (nTF), in which model decisions determined which records advanced. We measured sensitivity, specificity, positive and negative predictive values and F1 for each model, stage and review; interrater agreement (Cohen's and Fleiss's kappa); and within-model heterogeneity across reviews (Cochran's Q, I²). Discordant human–LLM decisions were audited, described and categorized. RESULTS Performance varied by context, stage and workflow. In nTF analyses, ChatGPT was more sensitive (76.7%–100% vs 75%–79.7% for Gemini) and Gemini more specific (82.7%–98.4% vs 83.2%–94.3% for ChatGPT). ChatGPT's sensitivity was heterogeneous across reviews (Q=15.91, p<.0005, I²=87%) whereas Gemini's was stable (I²≈0%); specificity was heterogeneous for both (I²=75% for ChatGPT and 98% for Gemini). Precision remained low throughout. Discordance analysis indicated systematic rather than random misclassification. Human–model agreement was modest to moderate (Cohen's kappa 0.255–0.442) and lower than inter-model agreement (up to 0.781). CONCLUSIONS LLMs can improve efficiency mainly as high-sensitivity assistants that reduce screening burden while leaving final adjudication to humans. Their safest role is within audited, human-guided hybrid workflows rather than fully autonomous pipelines, and potential efficiency gains should be formally quantified against possible losses in quality and completeness.