Conrad W. Safranek
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Conrad W. Safranek has a total of 53 co-authors. Here are the top co-authors by number of shared publications:
- David Chartash
Department of Biostatistics, School of Public Health, Yale University, New Haven, CT, USA
12 shared publications - Richard Andrew Taylor
Department of Emergency Medicine University of Virginia Charlottesville Virginia USA
12 shared publications - Aidan Gilson
Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, United States
9 shared publications - Vimig Socrates
Department of Biomedical Informatics and Data Science, School of Medicine, Yale University, New Haven, CT, United States
9 shared publications - Thomas Huang
School of Medicine, Yale University, New Haven, CT, United States
8 shared publications - Ling Chi
Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, United States
7 shared publications
Latest Publications
- Open Access
- 28 May 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12 (1) , e89996
Ambient AI Scribes to Create Educational Feedback Notes for Medical Students: Randomized Trial
Background: High-quality observation and feedback contribute to the development of clinical competence and professional growth in medical education. Faculty often struggle to translate verbal observations into written feedback because of documentation burden and competing demands. Ambient artificial intelligence (AI) scribes, already adopted in clinical practice, may address this challenge by capturing verbal exchanges and generating structured notes. Objective: The purpose of this study was to examine the use of ambient AI scribes to generate educational feedback notes during a formative medical interviewing workshop for first-year medical students in March and April 2025. Methods: Thirteen instructors were randomized to control (human-only) or intervention (AI scribe–assisted) workflows to complete narrative feedback forms. The intervention group used an AI scribe to generate transcripts of student-instructor encounters, which were then summarized into feedback notes using a large language model and edited by the instructors before submission. All narratives were scored using the Evaluation of Feedback Captured Tool (EFeCT). Factual accuracy of a subsample of unedited AI feedback summaries was reviewed against the source transcripts. Task load and usability were measured using NASA Task Load Index and System Usability Scale, respectively. Results: Instructors submitted feedback on 92.2% (94/102) of the students. EFeCT scores on the scale from 0 to 5 were higher for human-edited AI narratives (median 3.00, IQR 2.00-4.00) and unedited AI summaries (median 3.00, IQR 3.00-4.00) than for human-only narratives (median 2.00, IQR 1.75-3.00; P <.001). Human-only narratives were shorter than AI-assisted outputs ( P <.001). Review of 117 AI-generated feedback elements showed a 6.8% (n=8) mischaracterization and 1.7% (n=2) hallucination rate, with most errors corrected during editing. Task load was high, and usability was marginal in both the control and intervention groups, with no significant differences ( P =.31 and P =.40, respectively). Conclusions: An ambient AI scribe–assisted workflow improved the quality of written narrative feedback with no observed increase in instructor effort compared to human-only documentation. Although occasional inaccuracies required review, this innovation has the potential to transform feedback documentation.
- 28 May 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12 (1) , e89996
Open AccessAmbient AI Scribes to Create Educational Feedback Notes for Medical Students: Randomized Trial
Background: High-quality observation and feedback contribute to the development of clinical competence and professional growth in medical education. Faculty often struggle to translate verbal observations into written feedback because of documentation burden and competing demands. Ambient artificial intelligence (AI) scribes, already adopted in clinical practice, may address this challenge by capturing verbal exchanges and generating structured notes. Objective: The purpose of this study was to examine the use of ambient AI scribes to generate educational feedback notes during a formative medical interviewing workshop for first-year medical students in March and April 2025. Methods: Thirteen instructors were randomized to control (human-only) or intervention (AI scribe–assisted) workflows to complete narrative feedback forms. The intervention group used an AI scribe to generate transcripts of student-instructor encounters, which were then summarized into feedback notes using a large language model and edited by the instructors before submission. All narratives were scored using the Evaluation of Feedback Captured Tool (EFeCT). Factual accuracy of a subsample of unedited AI feedback summaries was reviewed against the source transcripts. Task load and usability were measured using NASA Task Load Index and System Usability Scale, respectively. Results: Instructors submitted feedback on 92.2% (94/102) of the students. EFeCT scores on the scale from 0 to 5 were higher for human-edited AI narratives (median 3.00, IQR 2.00-4.00) and unedited AI summaries (median 3.00, IQR 3.00-4.00) than for human-only narratives (median 2.00, IQR 1.75-3.00; P <.001). Human-only narratives were shorter than AI-assisted outputs ( P <.001). Review of 117 AI-generated feedback elements showed a 6.8% (n=8) mischaracterization and 1.7% (n=2) hallucination rate, with most errors corrected during editing. Task load was high, and usability was marginal in both the control and intervention groups, with no significant differences ( P =.31 and P =.40, respectively). Conclusions: An ambient AI scribe–assisted workflow improved the quality of written narrative feedback with no observed increase in instructor effort compared to human-only documentation. Although occasional inaccuracies required review, this innovation has the potential to transform feedback documentation.
- Open Access
- 13 March 2026
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5 (1) , e93250
Correction: Real-World Evidence Synthesis of Digital Scribes Using Ambient Listening and Generative Artificial Intelligence for Clinician Documentation Workflows: Rapid Review
- 13 March 2026
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5 (1) , e93250
Open AccessCorrection: Real-World Evidence Synthesis of Digital Scribes Using Ambient Listening and Generative Artificial Intelligence for Clinician Documentation Workflows: Rapid Review
- Top Cited PapersOpen Access
- 10 October 2025
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 4, e76743
Real-World Evidence Synthesis of Digital Scribes Using Ambient Listening and Generative Artificial Intelligence for Clinician Documentation Workflows: Rapid Review
Background: As physicians spend up to twice as much time on electronic health record tasks as on direct patient care, digital scribes have emerged as a promising solution to restore patient-clinician communication and reduce documentation burden—making it essential to study their real-world impact on clinical workflows, efficiency, and satisfaction. Objective: This study aimed to synthesize evidence on clinician efficiency, user satisfaction, quality, and practical barriers associated with the use of digital scribes using ambient listening and generative artificial intelligence (AI) in real-world clinical settings. Methods: A rapid review was conducted to evaluate the real-world evidence of digital scribes using ambient listening and generative AI in clinical practice from 2014 to 2024. Data were collected from Ovid MEDLINE, Embase, Web of Science–Core Collection, Cochrane CENTRAL and Reviews, and PubMed Central. Predefined eligibility criteria focused on studies addressing clinical implementation, excluding those centered solely on technical development or model validation. The findings of each study were synthesized and analyzed through the QUEST human evaluation framework for quality and safety and the Systems Engineering Initiative for Patient Safety (SEIPS) 3.0 model to assess integration into clinicians’ workflows and experience. Results: Of the 1450 studies identified, 6 met the inclusion criteria. These studies included an observational study, a case report, a peer-matched cohort study, and survey-based assessments conducted across academic health systems, community settings, and outpatient practices. The major themes noted were as follows: (1) they decreased self-reported documentation times, with associated increased length of notes; (2) physician burnout measured using standardized scales was unaffected, but physician engagement improved; (3) physician productivity, assessed via billing metrics, was unchanged; and (4) the studies fell short when compared to standardized frameworks. Conclusions: Digital scribes show promise in reducing documentation burden and enhancing clinician satisfaction, thereby supporting workflow efficiency. However, the currently available evidence is sparse. Future real-world, multifaceted studies are needed before AI scribes can be recommended unequivocally.
- 10 October 2025
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 4, e76743
Top Cited PapersOpen AccessReal-World Evidence Synthesis of Digital Scribes Using Ambient Listening and Generative Artificial Intelligence for Clinician Documentation Workflows: Rapid Review
Background: As physicians spend up to twice as much time on electronic health record tasks as on direct patient care, digital scribes have emerged as a promising solution to restore patient-clinician communication and reduce documentation burden—making it essential to study their real-world impact on clinical workflows, efficiency, and satisfaction. Objective: This study aimed to synthesize evidence on clinician efficiency, user satisfaction, quality, and practical barriers associated with the use of digital scribes using ambient listening and generative artificial intelligence (AI) in real-world clinical settings. Methods: A rapid review was conducted to evaluate the real-world evidence of digital scribes using ambient listening and generative AI in clinical practice from 2014 to 2024. Data were collected from Ovid MEDLINE, Embase, Web of Science–Core Collection, Cochrane CENTRAL and Reviews, and PubMed Central. Predefined eligibility criteria focused on studies addressing clinical implementation, excluding those centered solely on technical development or model validation. The findings of each study were synthesized and analyzed through the QUEST human evaluation framework for quality and safety and the Systems Engineering Initiative for Patient Safety (SEIPS) 3.0 model to assess integration into clinicians’ workflows and experience. Results: Of the 1450 studies identified, 6 met the inclusion criteria. These studies included an observational study, a case report, a peer-matched cohort study, and survey-based assessments conducted across academic health systems, community settings, and outpatient practices. The major themes noted were as follows: (1) they decreased self-reported documentation times, with associated increased length of notes; (2) physician burnout measured using standardized scales was unaffected, but physician engagement improved; (3) physician productivity, assessed via billing metrics, was unchanged; and (4) the studies fell short when compared to standardized frameworks. Conclusions: Digital scribes show promise in reducing documentation burden and enhancing clinician satisfaction, thereby supporting workflow efficiency. However, the currently available evidence is sparse. Future real-world, multifaceted studies are needed before AI scribes can be recommended unequivocally.
- Open Access
- 7 July 2025
- Published by American Academy of Child and Adolescent Psychiatry in JAACAP Connect
- Vol. 13 (1)
Harnessing Large Language Models to Promote Attainment of ACGME Competencies in Child Psychiatry
Artificial intelligence and large language models (LLMs) such as ChatGPT are spurring both enthusiasm and trepidation within the medical education community. Here, we explore the application of LLMs in medical education in child and adolescent psychiatry in pursuit of the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Graduate Medical Education. In particular, we demonstrate how one might leverage LLMs to achieve ACGME competencies in clinical skills in the major treatment modalities. We use the voice feature of the ChatGPT app to simulate a patient interaction during which the provider must engage in crisis intervention. We discuss the realistic nature of the scenario produced by ChatGPT, the ability of ChatGPT to serve as a clinical coach, and the opportunities afforded by prompt engineering. Interacting with LLMs throughout medical education in child psychiatry both promotes innovation and fosters an understanding of their limitations in clinical care. Fluency in the language of these models is of the utmost importance as their influence in health care continues to expand.
- 7 July 2025
- Published by American Academy of Child and Adolescent Psychiatry in JAACAP Connect
- Vol. 13 (1)
Open AccessHarnessing Large Language Models to Promote Attainment of ACGME Competencies in Child Psychiatry
Artificial intelligence and large language models (LLMs) such as ChatGPT are spurring both enthusiasm and trepidation within the medical education community. Here, we explore the application of LLMs in medical education in child and adolescent psychiatry in pursuit of the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Graduate Medical Education. In particular, we demonstrate how one might leverage LLMs to achieve ACGME competencies in clinical skills in the major treatment modalities. We use the voice feature of the ChatGPT app to simulate a patient interaction during which the provider must engage in crisis intervention. We discuss the realistic nature of the scenario produced by ChatGPT, the ability of ChatGPT to serve as a clinical coach, and the opportunities afforded by prompt engineering. Interacting with LLMs throughout medical education in child psychiatry both promotes innovation and fosters an understanding of their limitations in clinical care. Fluency in the language of these models is of the utmost importance as their influence in health care continues to expand.
- Open Access
- 29 April 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Real World Evidence-Synthesis of Digital Scribes using Ambient Listening and Generative Artificial Intelligence for Clinician Documentation Workflows: A Rapid Review (Preprint)
BACKGROUND Digital scribes using ambient listening and generative AI have the potential to streamline clinical documentation and enhance workflow efficiency. However, despite growing interest, real-world evidence on their effects on clinician efficiency, satisfaction, quality, and integration remains limited. OBJECTIVE To synthesize evidence on clinician efficiency, user satisfaction, quality, and practical barriers associated with the use of digital scribes employing ambient listening and generative artificial intelligence (AI) in real-world clinical settings. METHODS A rapid review was conducted to evaluate the real-world evidence of digital scribes using ambient listening and generative AI in clinical practice from 2014 to 2024. Data was collected from Ovid MEDLINE, Embase, Web of Science - Core Collection, Cochrane CENTRAL and Reviews, and PubMed Central. Predefined eligibility criteria focused on studies addressing clinical implementation, excluding those centered solely on technical development or model validation. The findings of each study were synthesized and analyzed through the QUEST human evaluation framework for quality and safety and the SEIPS 3.0 Model to assess integration into clinician’s workflows and experience. RESULTS Of the 1,450 studies identified, six met inclusion criteria. These studies included an observational study, a case report, a peer-matched cohort study, and survey-based assessments conducted across academic health systems, community settings, and outpatient practices. The major themes noted were: (1) They decreased self-reported documentation times, with associated increased length of notes, (2) Physician burnout measured using standardized scales was unaffected, but physician engagement improved, (3) Physician productivity, assessed via billing metrics, was unchanged, (4) the studies fell short when compared to standardized frameworks. CONCLUSIONS Digital scribes show promise in reducing documentation burden and enhancing clinician satisfaction, thereby supporting workflow efficiency. However, the current available evidence is sparse. Future real-world, multifaceted studies are needed before AI scribes can be recommended unequivocally. CLINICALTRIAL N/A
- 29 April 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessReal World Evidence-Synthesis of Digital Scribes using Ambient Listening and Generative Artificial Intelligence for Clinician Documentation Workflows: A Rapid Review (Preprint)
BACKGROUND Digital scribes using ambient listening and generative AI have the potential to streamline clinical documentation and enhance workflow efficiency. However, despite growing interest, real-world evidence on their effects on clinician efficiency, satisfaction, quality, and integration remains limited. OBJECTIVE To synthesize evidence on clinician efficiency, user satisfaction, quality, and practical barriers associated with the use of digital scribes employing ambient listening and generative artificial intelligence (AI) in real-world clinical settings. METHODS A rapid review was conducted to evaluate the real-world evidence of digital scribes using ambient listening and generative AI in clinical practice from 2014 to 2024. Data was collected from Ovid MEDLINE, Embase, Web of Science - Core Collection, Cochrane CENTRAL and Reviews, and PubMed Central. Predefined eligibility criteria focused on studies addressing clinical implementation, excluding those centered solely on technical development or model validation. The findings of each study were synthesized and analyzed through the QUEST human evaluation framework for quality and safety and the SEIPS 3.0 Model to assess integration into clinician’s workflows and experience. RESULTS Of the 1,450 studies identified, six met inclusion criteria. These studies included an observational study, a case report, a peer-matched cohort study, and survey-based assessments conducted across academic health systems, community settings, and outpatient practices. The major themes noted were: (1) They decreased self-reported documentation times, with associated increased length of notes, (2) Physician burnout measured using standardized scales was unaffected, but physician engagement improved, (3) Physician productivity, assessed via billing metrics, was unchanged, (4) the studies fell short when compared to standardized frameworks. CONCLUSIONS Digital scribes show promise in reducing documentation burden and enhancing clinician satisfaction, thereby supporting workflow efficiency. However, the current available evidence is sparse. Future real-world, multifaceted studies are needed before AI scribes can be recommended unequivocally. CLINICALTRIAL N/A
- 10 April 2025
- Published by Elsevier in Journal of the American Academy of Child & Adolescent Psychiatry
- Vol. 65 (2) , 139-142
Managing Screen Time in the Age of AI: Supporting Healthier Habits for Teens From the Inpatient Setting
- 10 April 2025
- Published by Elsevier in Journal of the American Academy of Child & Adolescent Psychiatry
- Vol. 65 (2) , 139-142
Managing Screen Time in the Age of AI: Supporting Healthier Habits for Teens From the Inpatient Setting
- Open Access
- 2 August 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e60336
Patient-Representing Population's Perceptions of GPT-Generated Versus Standard Emergency Department Discharge Instructions: Randomized Blind Survey Assessment
Background: Discharge instructions are a key form of documentation and patient communication in the time of transition from the emergency department (ED) to home. Discharge instructions are time-consuming and often underprioritized, especially in the ED, leading to discharge delays and possibly impersonal patient instructions. Generative artificial intelligence and large language models (LLMs) offer promising methods of creating high-quality and personalized discharge instructions; however, there exists a gap in understanding patient perspectives of LLM-generated discharge instructions. Objective: We aimed to assess the use of LLMs such as ChatGPT in synthesizing accurate and patient-accessible discharge instructions in the ED. Methods: We synthesized 5 unique, fictional ED encounters to emulate real ED encounters that included a diverse set of clinician history, physical notes, and nursing notes. These were passed to GPT-4 in Azure OpenAI Service (Microsoft) to generate LLM-generated discharge instructions. Standard discharge instructions were also generated for each of the 5 unique ED encounters. All GPT-generated and standard discharge instructions were then formatted into standardized after-visit summary documents. These after-visit summaries containing either GPT-generated or standard discharge instructions were randomly and blindly administered to Amazon MTurk respondents representing patient populations through Amazon MTurk Survey Distribution. Discharge instructions were assessed based on metrics of interpretability of significance, understandability, and satisfaction. Results: Our findings revealed that survey respondents’ perspectives regarding GPT-generated and standard discharge instructions were significantly (P=.01) more favorable toward GPT-generated return precautions, and all other sections were considered noninferior to standard discharge instructions. Of the 156 survey respondents, GPT-generated discharge instructions were assigned favorable ratings, “agree” and “strongly agree,” more frequently along the metric of interpretability of significance in discharge instruction subsections regarding diagnosis, procedures, treatment, post-ED medications or any changes to medications, and return precautions. Survey respondents found GPT-generated instructions to be more understandable when rating procedures, treatment, post-ED medications or medication changes, post-ED follow-up, and return precautions. Satisfaction with GPT-generated discharge instruction subsections was the most favorable in procedures, treatment, post-ED medications or medication changes, and return precautions. Wilcoxon rank-sum test of Likert responses revealed significant differences (P=.01) in the interpretability of significant return precautions in GPT-generated discharge instructions compared to standard discharge instructions but not for other evaluation metrics and discharge instruction subsections. Conclusions: This study demonstrates the potential for LLMs such as ChatGPT to act as a method of augmenting current documentation workflows in the ED to reduce the documentation burden of physicians. The ability of LLMs to provide tailored instructions for patients by improving readability and making instructions more applicable to patients could improve upon the methods of communication that currently exist.
- 2 August 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e60336
Open AccessPatient-Representing Population's Perceptions of GPT-Generated Versus Standard Emergency Department Discharge Instructions: Randomized Blind Survey Assessment
Background: Discharge instructions are a key form of documentation and patient communication in the time of transition from the emergency department (ED) to home. Discharge instructions are time-consuming and often underprioritized, especially in the ED, leading to discharge delays and possibly impersonal patient instructions. Generative artificial intelligence and large language models (LLMs) offer promising methods of creating high-quality and personalized discharge instructions; however, there exists a gap in understanding patient perspectives of LLM-generated discharge instructions. Objective: We aimed to assess the use of LLMs such as ChatGPT in synthesizing accurate and patient-accessible discharge instructions in the ED. Methods: We synthesized 5 unique, fictional ED encounters to emulate real ED encounters that included a diverse set of clinician history, physical notes, and nursing notes. These were passed to GPT-4 in Azure OpenAI Service (Microsoft) to generate LLM-generated discharge instructions. Standard discharge instructions were also generated for each of the 5 unique ED encounters. All GPT-generated and standard discharge instructions were then formatted into standardized after-visit summary documents. These after-visit summaries containing either GPT-generated or standard discharge instructions were randomly and blindly administered to Amazon MTurk respondents representing patient populations through Amazon MTurk Survey Distribution. Discharge instructions were assessed based on metrics of interpretability of significance, understandability, and satisfaction. Results: Our findings revealed that survey respondents’ perspectives regarding GPT-generated and standard discharge instructions were significantly (P=.01) more favorable toward GPT-generated return precautions, and all other sections were considered noninferior to standard discharge instructions. Of the 156 survey respondents, GPT-generated discharge instructions were assigned favorable ratings, “agree” and “strongly agree,” more frequently along the metric of interpretability of significance in discharge instruction subsections regarding diagnosis, procedures, treatment, post-ED medications or any changes to medications, and return precautions. Survey respondents found GPT-generated instructions to be more understandable when rating procedures, treatment, post-ED medications or medication changes, post-ED follow-up, and return precautions. Satisfaction with GPT-generated discharge instruction subsections was the most favorable in procedures, treatment, post-ED medications or medication changes, and return precautions. Wilcoxon rank-sum test of Likert responses revealed significant differences (P=.01) in the interpretability of significant return precautions in GPT-generated discharge instructions compared to standard discharge instructions but not for other evaluation metrics and discharge instruction subsections. Conclusions: This study demonstrates the potential for LLMs such as ChatGPT to act as a method of augmenting current documentation workflows in the ED to reduce the documentation burden of physicians. The ability of LLMs to provide tailored instructions for patients by improving readability and making instructions more applicable to patients could improve upon the methods of communication that currently exist.
- Open Access
- 8 May 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Patient-Representing Population's Perceptions of GPT-Generated Versus Standard Emergency Department Discharge Instructions: Randomized Blind Survey Assessment (Preprint)
BACKGROUND Discharge instructions are a key form of documentation and patient communication in the time of transition from the emergency department (ED) to home. Discharge instructions are time-consuming and often underprioritized, especially in the ED, leading to discharge delays and possibly impersonal patient instructions. Generative artificial intelligence and large language models (LLMs) offer promising methods of creating high-quality and personalized discharge instructions; however, there exists a gap in understanding patient perspectives of LLM-generated discharge instructions. OBJECTIVE We aimed to assess the use of LLMs such as ChatGPT in synthesizing accurate and patient-accessible discharge instructions in the ED. METHODS We synthesized 5 unique, fictional ED encounters to emulate real ED encounters that included a diverse set of clinician history, physical notes, and nursing notes. These were passed to GPT-4 in Azure OpenAI Service (Microsoft) to generate LLM-generated discharge instructions. Standard discharge instructions were also generated for each of the 5 unique ED encounters. All GPT-generated and standard discharge instructions were then formatted into standardized after-visit summary documents. These after-visit summaries containing either GPT-generated or standard discharge instructions were randomly and blindly administered to Amazon MTurk respondents representing patient populations through Amazon MTurk Survey Distribution. Discharge instructions were assessed based on metrics of interpretability of significance, understandability, and satisfaction. RESULTS Our findings revealed that survey respondents’ perspectives regarding GPT-generated and standard discharge instructions were significantly (P=.01) more favorable toward GPT-generated return precautions, and all other sections were considered noninferior to standard discharge instructions. Of the 156 survey respondents, GPT-generated discharge instructions were assigned favorable ratings, “agree” and “strongly agree,” more frequently along the metric of interpretability of significance in discharge instruction subsections regarding diagnosis, procedures, treatment, post-ED medications or any changes to medications, and return precautions. Survey respondents found GPT-generated instructions to be more understandable when rating procedures, treatment, post-ED medications or medication changes, post-ED follow-up, and return precautions. Satisfaction with GPT-generated discharge instruction subsections was the most favorable in procedures, treatment, post-ED medications or medication changes, and return precautions. Wilcoxon rank-sum test of Likert responses revealed significant differences (P=.01) in the interpretability of significant return precautions in GPT-generated discharge instructions compared to standard discharge instructions but not for other evaluation metrics and discharge instruction subsections. CONCLUSIONS This study demonstrates the potential for LLMs such as ChatGPT to act as a method of augmenting current documentation workflows in the ED to reduce the documentation burden of physicians. The ability of LLMs to provide tailored instructions for patients by improving readability and making instructions more applicable to patients could improve upon the methods of communication that currently exist.
- 8 May 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessPatient-Representing Population's Perceptions of GPT-Generated Versus Standard Emergency Department Discharge Instructions: Randomized Blind Survey Assessment (Preprint)
BACKGROUND Discharge instructions are a key form of documentation and patient communication in the time of transition from the emergency department (ED) to home. Discharge instructions are time-consuming and often underprioritized, especially in the ED, leading to discharge delays and possibly impersonal patient instructions. Generative artificial intelligence and large language models (LLMs) offer promising methods of creating high-quality and personalized discharge instructions; however, there exists a gap in understanding patient perspectives of LLM-generated discharge instructions. OBJECTIVE We aimed to assess the use of LLMs such as ChatGPT in synthesizing accurate and patient-accessible discharge instructions in the ED. METHODS We synthesized 5 unique, fictional ED encounters to emulate real ED encounters that included a diverse set of clinician history, physical notes, and nursing notes. These were passed to GPT-4 in Azure OpenAI Service (Microsoft) to generate LLM-generated discharge instructions. Standard discharge instructions were also generated for each of the 5 unique ED encounters. All GPT-generated and standard discharge instructions were then formatted into standardized after-visit summary documents. These after-visit summaries containing either GPT-generated or standard discharge instructions were randomly and blindly administered to Amazon MTurk respondents representing patient populations through Amazon MTurk Survey Distribution. Discharge instructions were assessed based on metrics of interpretability of significance, understandability, and satisfaction. RESULTS Our findings revealed that survey respondents’ perspectives regarding GPT-generated and standard discharge instructions were significantly (P=.01) more favorable toward GPT-generated return precautions, and all other sections were considered noninferior to standard discharge instructions. Of the 156 survey respondents, GPT-generated discharge instructions were assigned favorable ratings, “agree” and “strongly agree,” more frequently along the metric of interpretability of significance in discharge instruction subsections regarding diagnosis, procedures, treatment, post-ED medications or any changes to medications, and return precautions. Survey respondents found GPT-generated instructions to be more understandable when rating procedures, treatment, post-ED medications or medication changes, post-ED follow-up, and return precautions. Satisfaction with GPT-generated discharge instruction subsections was the most favorable in procedures, treatment, post-ED medications or medication changes, and return precautions. Wilcoxon rank-sum test of Likert responses revealed significant differences (P=.01) in the interpretability of significant return precautions in GPT-generated discharge instructions compared to standard discharge instructions but not for other evaluation metrics and discharge instruction subsections. CONCLUSIONS This study demonstrates the potential for LLMs such as ChatGPT to act as a method of augmenting current documentation workflows in the ED to reduce the documentation burden of physicians. The ability of LLMs to provide tailored instructions for patients by improving readability and making instructions more applicable to patients could improve upon the methods of communication that currently exist.
- 15 March 2024
- Published by Wiley in Otolaryngology–Head and Neck Surgery
- Vol. 170 (6) , 1512-1518
ChatGPT for Automated Cross‐Checking of Authors' Conflicts of Interest Against Industry Payments
Objective: The Centers for Medicare & Medicaid Services “OpenPayments” database tracks industry payments to US physicians to improve research conflicts of interest (COIs) transparency, but manual cross‐checking of articles' authors against this database is labor‐intensive. This study aims to assess the potential of large language models (LLMs) like ChatGPT to automate COI data analysis in medical publications.Study Design: An observational study analyzing the accuracy of ChatGPT in automating the cross‐checking of COI disclosures in medical research articles against the OpenPayments database.Setting: Publications regarding Food and Drug Administration‐approved biologics for chronic rhinosinusitis with nasal polyposis: omalizumab, mepolizumab, and dupilumab.Methods: First, ChatGPT evaluated author affiliations from PubMed to identify those based in the United States. Second, for author names matching 1 or multiple payment recipients in OpenPayments, ChatGPT undertook a comparative analysis between author affiliation and OpenPayments recipient metadata. Third, ChatGPT scrutinized full article COI statements, producing an intricate matrix of disclosures for each author against each relevant company (Sanofi, Regeneron, Genentech, Novartis, and GlaxoSmithKline). A random subset of responses was manually checked for accuracy.Results: In total, 78 relevant articles and 294 unique US authors were included, leading to 980 LLM queries. Manual verification showed accuracies of 100% (200/200; 95% confidence interval [CI]: 98.1%‐100%) for country analysis, 97.4% (113/116; 95% CI: 92.7%‐99.1%) for matching author affiliations with OpenPayments metadata, and 99.2% (1091/1100; 95% CI: 98.5%‐99.6%) for COI statement data extraction.Conclusion: LLMs have robust potential to automate author‐company‐specific COI cross‐checking against the OpenPayments database. Our findings pave the way for streamlined, efficient, and accurate COI assessment that could be widely employed across medical research.
- 15 March 2024
- Published by Wiley in Otolaryngology–Head and Neck Surgery
- Vol. 170 (6) , 1512-1518
ChatGPT for Automated Cross‐Checking of Authors' Conflicts of Interest Against Industry Payments
Objective: The Centers for Medicare & Medicaid Services “OpenPayments” database tracks industry payments to US physicians to improve research conflicts of interest (COIs) transparency, but manual cross‐checking of articles' authors against this database is labor‐intensive. This study aims to assess the potential of large language models (LLMs) like ChatGPT to automate COI data analysis in medical publications.Study Design: An observational study analyzing the accuracy of ChatGPT in automating the cross‐checking of COI disclosures in medical research articles against the OpenPayments database.Setting: Publications regarding Food and Drug Administration‐approved biologics for chronic rhinosinusitis with nasal polyposis: omalizumab, mepolizumab, and dupilumab.Methods: First, ChatGPT evaluated author affiliations from PubMed to identify those based in the United States. Second, for author names matching 1 or multiple payment recipients in OpenPayments, ChatGPT undertook a comparative analysis between author affiliation and OpenPayments recipient metadata. Third, ChatGPT scrutinized full article COI statements, producing an intricate matrix of disclosures for each author against each relevant company (Sanofi, Regeneron, Genentech, Novartis, and GlaxoSmithKline). A random subset of responses was manually checked for accuracy.Results: In total, 78 relevant articles and 294 unique US authors were included, leading to 980 LLM queries. Manual verification showed accuracies of 100% (200/200; 95% confidence interval [CI]: 98.1%‐100%) for country analysis, 97.4% (113/116; 95% CI: 92.7%‐99.1%) for matching author affiliations with OpenPayments metadata, and 99.2% (1091/1100; 95% CI: 98.5%‐99.6%) for COI statement data extraction.Conclusion: LLMs have robust potential to automate author‐company‐specific COI cross‐checking against the OpenPayments database. Our findings pave the way for streamlined, efficient, and accurate COI assessment that could be widely employed across medical research.
- Open Access
- 13 March 2024
- Published by Elsevier in JACEP Open
- Vol. 5 (2) , e13133
Automated HEART score determination via ChatGPT: Honing a framework for iterative prompt development
- 13 March 2024
- Published by Elsevier in JACEP Open
- Vol. 5 (2) , e13133
Open AccessAutomated HEART score determination via ChatGPT: Honing a framework for iterative prompt development