Evaluation of Large Language Models for Peer Review in Transplantation Research: Algorithm Validation Study
Top Cited Papers
Open Access
- 11 February 2026
- journal article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e84322
Abstract
Background: Peer review remains central to ensuring research quality, yet it is constrained by reviewer fatigue and human bias. The rapid rise in scientific publishing has worsened these challenges, prompting interest in whether large language models (LLMs) can support or improve the peer review process. Objective: This study aimed to address critical gaps in the use of LLMs for peer review of papers in the field of organ transplantation by (1) comparing the performance of 5 recent open-source LLMs; (2) evaluating the impact of author affiliations—prestigious, less prestigious, and none—on LLM review outcomes; and (3) examining the influence of prompt engineering strategies, including zero-shot prompting, few-shot prompting, tree of thoughts (ToT) prompting, and retrieval-augmented generation (RAG), on review decisions. Methods: A dataset of 200 transplantation papers published between 2024 and 2025 across 4 journal quartiles was evaluated using 5 state-of-the-art open-source LLMs (Llama 3.3, Mistral 7B, Gemma 2, DeepSeek r1-distill Qwen, and Qwen 2.5). The 4 prompting techniques (zero-shot prompting, few-shot prompting, ToT prompting, and RAG) were tested under multiple temperature settings. Models were instructed to categorize papers into quartiles. To assess fairness, each paper was evaluated 3 times: with no affiliation, a prestigious affiliation, and a less prestigious affiliation. Accuracy, decisions, runtime, and computing resource use were recorded. Chi-square tests and adjusted Pearson residuals were used to examine the presence of affiliation bias. Results: RAG with a temperature of 0.5 achieved the best overall performance (exact match accuracy: 0.35; loose match accuracy: 0.78). Across all models, LLMs frequently assigned manuscripts to quartile 2 and quartile 3 while avoiding extreme quartiles (quartile 1 and quartile 4). None of the models demonstrated affiliation bias, though Gemma 2 (P=.08) and Qwen 2.5 (P=.054) were substantially biased. Each model displayed unique “personalities” in quartile predictions, influencing consistency. Mistral had the highest exact match accuracy (0.35) despite having both the lowest average runtime (1246.378 seconds) and computing resource use (7 billion parameters). While accuracy was insufficient for independent review, LLMs showed value in supporting preliminary triage tasks. Conclusions: Current open-source LLMs are not reliable enough to replace human peer reviewers. The largely absent affiliation bias suggests potential advantages in fairness, but these benefits do not offset the low decision accuracy. Mistral demonstrated the greatest accuracy and computational efficiency, and RAG with a moderate temperature emerged as the most effective prompting strategy. If LLMs are used to assist in peer review, their outputs require nonnegotiable human supervision to ensure correct judgment and appropriate editorial decisions.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- The Impact of Generative Artificial Intelligence on the External Review of Scientific Manuscripts and Editorial Peer Review ProcessesThe American Journal of Pathology, 2024
- ChatGPT‐4 Performs Clinical Information Retrieval Tasks Using Consistently More Trustworthy Resources Than Does Google Search for Queries Concerning the Latarjet ProcedureArthroscopy, 2024
- Where have all the reviewers gone?Advances in Health Sciences Education, 2024
- Is ChatGPT corrupting peer review? Telltale words hint at AI useNature, 2024
- Affiliation Bias in Peer Review of Abstracts by a Large Language ModelJAMA, 2024
- Evaluating the Application of Large Language Models in Clinical Research ContextsJAMA Network Open, 2023
- Stop the peer-review treadmill. I want to get offNature, 2023
- What ChatGPT and generative AI mean for scienceNature, 2023
- Effect of Blinded Peer Review on Abstract AcceptanceJAMA, 2006
- Effect of Institutional Prestige on Reviewers' Recommendations and Editorial DecisionsJAMA, 1994