Usage of and Satisfaction with Artificial Intelligence-Generated Draft Replies to Patient Portal Messages
- 28 May 2026
- journal article
- research article
- Published by Georg Thieme Verlag KG in Applied Clinical Informatics
- Vol. 17 (03) , 510-517
Abstract
This study aimed to describe the usage and perceptions of physicians and nurses in primary and specialty care on the acceptability, usability, and quality of artificial intelligence (AI)-generated draft replies to patient portal messages. We conducted a pilot study of 80 users across seven clinics within a large academic health system in Pennsylvania. Epic's Augmented Response Technology (ART) used a large language model to present clinicians with draft replies to patient messages. To measure usage, we calculated the frequency with which replies were initiated with ART-generated drafts, as well as the overlap between drafts and replies sent to patients. To measure acceptability, usability, and quality, we developed and disseminated a 15-item survey. Voluntary pilot participants included 80 users (36 physicians, 25 nurses [registered nurses/licensed practice nurses], 10 nurse practitioners, 9 medical assistants). The mean rate of starting with an ART draft was 20.2%, and 40% of replies exhibited little to no editing of the draft. Of the 52 survey respondents, 66% (n = 33) agreed that AI-generated drafts were useful, and 46% (n = 23) agreed that ART improved the quality of replies to patient messages. Across survey questions, a higher percentage of nurses reported favorable opinions of the technology than did physicians. In free-response questions, the most common themes were that the message content was helpful (28.3%, n = 13) and that it reduced perceived effort or time (23.9%, n = 11), but that drafts occasionally contained incorrect or inappropriate content (34.9%, n = 15) or too often suggested an appointment (27.9%, n = 12). In this voluntary pilot study, AI-generated draft replies to patient portal messages were generally rated as acceptable and useful, with nurses and primary care clinicians more often reporting acceptance of the technology than physicians or specialty care clinicians. Scaling efforts should continue to monitor acceptability and effects on cognitive burden, in addition to safety.Keywords
This publication has 31 references indexed in Scilit:
- Ethics in Patient Preferences for Artificial Intelligence–Drafted Responses to Electronic MessagesJAMA Network Open, 2025
- Utility of Artificial Intelligence–Generative Draft Replies to Patient MessagesJAMA Network Open, 2024
- Are Artificial Intelligence–Generated Replies the Answer to the Electronic Health Record Inbox Problem?JAMA Network Open, 2024
- Large Language Model–Based Responses to Patients’ In-Basket MessagesJAMA Network Open, 2024
- AI-Generated Draft Replies Integrated Into Health Records and Physicians’ Electronic CommunicationJAMA Network Open, 2024
- Artificial Intelligence–Generated Draft Replies to Patient Inbox MessagesJAMA Network Open, 2024
- The Electronic Inbox—Benefits, Questions, and Solutions for the Road AheadJAMA, 2023
- Death by Patient PortalJAMA, 2023
- Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media ForumJAMA Internal Medicine, 2023
- Trends in Electronic Health Record Inbox Messaging During the COVID-19 Pandemic in an Ambulatory Practice Network in New EnglandJAMA Network Open, 2021