Conversational Assistants to support Heart Failure Patients: comparing a Neurosymbolic Architecture with ChatGPT
Preprint
Abstract
Conversational assistants are becoming more and more popular, including in healthcare, partly because of the availability and capabilities of Large Language Models. There is a need for controlled, probing evaluations with real stakeholders which can highlight advantages and disadvantages of more traditional architectures and those based on generative AI. We present a within-group user study to compare two versions of a conversational assistant that allows heart failure patients to ask about salt content in food. One version of the system was developed in-house with a neurosymbolic architecture, and one is based on ChatGPT. The evaluation shows that the in-house system is more accurate, completes more tasks and is less verbose than the one based on ChatGPT; on the other hand, the one based on ChatGPT makes fewer speech errors and requires fewer clarifications to complete the task. Patients show no preference for one over the other.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- PaniniQA: Enhancing Patient Education Through Interactive Question AnsweringTransactions of the Association for Computational Linguistics, 2023
- Learnings From Pilot Testing HFChat, An Interactive Dialogue Agent That Can Assist Black/African American And Hispanic/Latino Patients With Heart FailureJournal of Cardiac Failure, 2023
- Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System2022
- The AI Doctor Is In: A Survey of Task-Oriented Dialogue Systems for Healthcare Applications2022
- “That's (not) the output I expected!” On the role of end user expectations in creating explanations of AI systemsArtificial Intelligence, 2021
- Soloist: BuildingTask Bots at Scale with Transfer Learning and Machine TeachingTransactions of the Association for Computational Linguistics, 2021
- A Wizard-of-Oz Interface and Persona-based Methodology for Collecting Health Counseling Dialog2020
- Alexa as Coach: Leveraging Smart Speakers to Build Social Agents that Reduce Public Speaking Anxiety2020
- Are we teaching what patients need to know? Building skills in heart failure self-careHeart & Lung, 2009
- Evaluating spoken dialogue agents with PARADISE: Two case studiesComputer Speech & Language, 1998