Evaluating a Large Language Model’s Ability to Answer Clinicians’ Requests for Evidence Summaries
Preprint
Abstract
Objective This study investigated the performance of a generative artificial intelligence (AI) tool using GPT-4 in answering clinical questions in comparison with medical librarians’ gold-standard evidence syntheses. Methods Questions were extracted from an in-house database of clinical evidence requests previously answered by medical librarians. Questions with multiple parts were subdivided into individual topics. A standardized prompt was developed using the COSTAR framework. Librarians submitted each question into aiChat, an internally-managed chat tool using GPT-4, and recorded the responses. The summaries generated by aiChat were evaluated on whether they contained the critical elements used in the established gold-standard summary of the librarian. A subset of questions was randomly selected for verification of references provided by aiChat. Results Of the 216 evaluated questions, aiChat’s response was assessed as “correct” for 180 (83.3%) questions, “partially correct” for 35 (16.2%) questions, and “incorrect” for 1 (0.5%) question. No significant differences were observed in question ratings by question category (p=0.39). For a subset of 30% (n=66) of questions, 162 references were provided in the aiChat summaries, and 60 (37%) were confirmed as nonfabricated. Conclusions Overall, the performance of a generative AI tool was promising. However, many included references could not be independently verified, and attempts were not made to assess whether any additional concepts introduced by aiChat were factually accurate. Thus, we envision this being the first of a series of investigations designed to further our understanding of how current and future versions of generative AI can be used and integrated into medical librarians’ workflow.Keywords
All Related Versions
- Published version: Journal of the Medical Library Association, 113 (1), 65.
This publication has 47 references indexed in Scilit:
- Reporting Use of AI in Research and Scholarly Publication—JAMA Network GuidanceJAMA, 2024
- Does ChatGPT Answer Otolaryngology Questions Accurately?The Laryngoscope, 2024
- Leveraging artificial intelligence in neurosurgery—unveiling ChatGPT for neurosurgical discharge summaries and operative reportsActa Neurochirurgica, 2024
- Accuracy and reproducibility of ChatGPT's free version answers about endometriosisInternational Journal of Gynecology & Obstetrics, 2023
- Accuracy and Reliability of Chatbot Responses to Physician QuestionsJAMA Network Open, 2023
- Is ChatGPT‐4 Accurate in Proofread a Manuscript in Otolaryngology–Head and Neck Surgery?Otolaryngology–Head and Neck Surgery, 2023
- Creation and Adoption of Large Language Models in MedicineJAMA, 2023
- Prompt Engineering with ChatGPT: A Guide for Academic WritersAnnals of Biomedical Engineering, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Harnessing the Promise of Artificial Intelligence ResponsiblyJAMA, 2023