Evaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study (Preprint)
Authors:
- Lea HerschbachLHLea Herschbach
- Teresa Festl-WietekTFTeresa Festl-Wietek
- Christian Stegemann-PhilippsCSChristian Stegemann-Philipps
- Alessandra SonaniniASAlessandra Sonanini
- Bernd HerrmannBHBernd Herrmann
- Rebecca ErschensRERebecca Erschens
- Anne Herrmann-WernerAHAnne Herrmann-Werner
- Friederike HolderriedFHFriederike Holderried
Preprint
- 22 August 2025
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)–based tools offer promising support for practicing and improving communication techniques among physicians. OBJECTIVE This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice? METHODS The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice. RESULTS The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice. CONCLUSIONS The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 27 (1), e82818.
This publication has 41 references indexed in Scilit:
- Ärztliche Gesprächsführung2020
- Use of simulated patients to develop communication skills in nursing education: An integrative reviewNurse Education Today, 2017
- A Comparison of Natural Language Processing Methods for Automated Coding of Motivational InterviewingJournal of Substance Abuse Treatment, 2016
- The Positive Effect of Immediate Feedback on Medical Student Education During the Surgical ClerkshipJournal of Surgical Education, 2014
- Psychotherapie in der Psychosomatik – Trends in Diagnostik und TherapieZeitschrift für Psychiatrie, Psychologie und Psychotherapie, 2012
- Good communication in psychiatry – a conceptual reviewEuropean Psychiatry, 2011
- Can general practitioners identify people with distress and mild depression? A meta-analysis of clinical accuracyJournal of Affective Disorders, 2011
- How does communication heal? Pathways linking clinician–patient communication to health outcomesPatient Education and Counseling, 2009
- Feedback: An essential element of student learning in clinical practiceNurse Education in Practice, 2008
- What is Motivational Interviewing?Behavioural and Cognitive Psychotherapy, 1995