The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation
Authors:
- Carrie YeCYCarrie Ye
- Joseph Ross MitchellJMJoseph Ross Mitchell
- Daniel C BaumgartDBDaniel C Baumgart
- Zechen MaZMZechen Ma
- Angela Lim FungAFAngela Lim Fung
- Daniela Garcia OrellanaDODaniela Garcia Orellana
- Juel ChowdhuryJCJuel Chowdhury
- Abdullah AbassAAAbdullah Abass
- Steven KatzSKSteven Katz
- Jacob L JaremkoJJJacob L Jaremko
- Pierre BoulangerPierre Boulanger
- Claire E H BarberCBClaire E H Barber
Open Access
- 8 April 2026
- journal article
- research article
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 28 (1) , e87057
Abstract
Background: Despite the transformative potential of large language models (LLMs) in health care, the rapid development of these tools has outpaced their rigorous evaluation. While artificial intelligence–specific reporting guidelines have been developed to address standardized reporting of artificial intelligence studies, there is currently no specific tool available for risk of bias assessment of LLM question-answer (QA) studies. Existing risk-of-bias tools for medical research are not well suited to the unique challenges of evaluating LLM-QA studies, which creates a critical gap in assessing their safety and effectiveness. Objective: This study aims to develop the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for LLM-QA studies to systematically evaluate the validity and risk of bias in LLM-QA studies. Methods: We conducted 2 literature reviews. The first was on quality assessment tools for LLM-QA studies, and the second was on LLM-QA studies, which informed the first draft of the AQAT:RoB. The draft AQAT:ROB was further refined through a prespecified iterative process of modified Delphi, consensus meeting, and validation. The first Delphi process occurred between May 1 and May 20, 2025, and the first consensus meeting was held on May 22. The first round of validation was completed by 4 evaluators, who were not part of the consensus meeting, on 16 randomly selected studies. As this first round of validation surpassed our a priori threshold of ≥80% agreement and a Cohen κ of ≥0.61 between evaluators, no further rounds of development and validation were undertaken. A second Delphi process occurred between February 20 and February 23, 2026, to vote on postpilot changes in response to peer review. Results: The AQAT:RoB consists of 5 high-level domains (Questions, Reference Answers, LLM Answers, Evaluators, Outcomes). These domains are subdivided into 9 subdomains. Each subdomain includes at least one “Support for Judgment” and at least one “Type of Bias” and is to be rated “low,” “high,” or “unclear” for risk of bias. A pilot evaluation was completed by internal validators who were not part of the consensus discussion and were asked to complete the AQAT:RoB form for each assigned study. Each of the 16 studies was evaluated by 2 evaluators independently. Pilot validation showed a percent agreement of 86.1% and a Cohen κ of 0.70 between assessors. Conclusions: The AQAT:RoB demonstrates promising initial reliability for assessing the validity or risk of bias in LLM-QA studies. The tool will benefit from future refinements, external validation, and periodic updates to keep pace with evolving technology.Keywords
This publication has 40 references indexed in Scilit:
- Large Language Models for Chatbot Health Advice StudiesJAMA Network Open, 2025
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- Examining the Performance of ChatGPT 3.5 and Microsoft Copilot in Otolaryngology: A Comparative Study with Otolaryngologists’ EvaluationIndian Journal of Otolaryngology and Head & Neck Surgery, 2024
- Relevance of medical information obtained from ChatGPT: Are large language models friends or foes?Archives of Cardiovascular Diseases, 2023
- APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision SupportJAMA Network Open, 2023
- Creation and Adoption of Large Language Models in MedicineJAMA, 2023
- The utility of the ChatGPT artificial intelligence tool for patient education and enquiry in robotic radical prostatectomyInternational Urology and Nephrology, 2023
- Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care QuestionsJAMA Network Open, 2023
- Dr. ChatGPT will see you now: How do Google and ChatGPT compare in answering patient questions on breast reconstruction?Journal of Plastic, Reconstructive & Aesthetic Surgery, 2023
- The utility of ChatGPT for cancer treatment informationmedRxiv, 2023