Artificial Intelligence–Supported Development of Health Guideline Questions
- 1 November 2024
- journal article
- research article
- Published by American College of Physicians in Annals of Internal Medicine
- Vol. 177 (11) , 1518-1529
Abstract
Guideline questions are typically proposed by experts. To assess how large language models (LLMs) can support the development of guideline questions, providing insights on approaches and lessons learned. Two approaches for guideline question generation were assessed: 1) identification of questions conveyed by online search queries and 2) direct generation of guideline questions by LLMs. For the former, the researchers retrieved popular queries on allergic rhinitis using Google Trends (GT) and identified those conveying questions using both manual and LLM-based methods. They then manually structured as guideline questions the queries that conveyed relevant questions. For the second approach, they tasked an LLM with proposing guideline questions, assuming the role of either a patient or a clinician. Allergic Rhinitis and its Impact on Asthma (ARIA) 2024 guidelines. None. Frequency of relevant questions generated. The authors retrieved 3975 unique queries using GT. From these, they identified 37 questions, of which 22 had not been previously posed by guideline panel members and 2 were eventually prioritized by the panel. Direct interactions with LLMs resulted in the generation of 22 unique relevant questions (11 not previously suggested by panel members), and 4 were eventually prioritized by the panel. In total, 6 of 39 final questions prioritized for the 2024 ARIA guidelines were not initially thought of by the panel. The researchers provide a set of practical insights on the implementation of their approaches based on the lessons learned. Single case study (ARIA guidelines). Approaches using LLMs can support the development of guideline questions, complementing traditional methods and potentially augmenting questions prioritized by guideline panels. Fraunhofer Cluster of Excellence for Immune-Mediated Diseases.Keywords
This publication has 16 references indexed in Scilit:
- Trends in Phenylephrine and Pseudoephedrine Sales in the USJAMA, 2024
- Large language models show human-like content biases in transmission chain experimentsProceedings of the National Academy of Sciences of the United States of America, 2023
- Limitations of large language models in medical applicationsPostgraduate Medical Journal, 2023
- Ethics of large language models in medicine and medical researchThe Lancet Digital Health, 2023
- Real‐world data using mHealth apps in rhinitis, rhinosinusitis and their multimorbiditiesClinical and Translational Allergy, 2022
- How to Interpret and Use a Clinical Practice Guideline or RecommendationJAMA, 2021
- Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines—2016 revisionJournal of Allergy and Clinical Immunology, 2017
- Infodemiology and InfoveillanceAmerican Journal of Preventive Medicine, 2011
- Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines: 2010 RevisionJournal of Allergy and Clinical Immunology, 2010
- Allergic Rhinitis and Its Impact on AsthmaJournal of Allergy and Clinical Immunology, 2001