Evaluating Human-AI Hybrid Conversational Systems with Chatbot Message Suggestions
Authors:
Open Access
- 26 October 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 534-544
Abstract
AI chatbots can offer suggestions to help humans answer questions by reducing text entry effort and providing relevant knowledge for unfamiliar questions. We study whether chatbot suggestions can help people answer knowledge-demanding questions in a conversation and influence response quality and efficiency. We conducted a large-scale crowdsourcing user study and evaluated 20 hybrid system variants and a human-only baseline. The hybrid systems used four chatbots of varied response quality and differed in the number of suggestions and whether to preset the message box with top suggestions. Experimental results show that chatbot suggestions---even using poor-performing chatbots---have consistently improved response efficiency. Compared with the human-only setting, hybrid systems have reduced response time by 12%--35% and keystrokes by 33%--60%, and users have adopted a suggestion for the final response without any changes in 44%--68% of the cases. In contrast, crowd workers in the human-only setting typed most of the response texts and copied 5% of the answers from other sites. However, we also found that chatbot suggestions did not always help response quality. Specifically, in hybrid systems equipped with poor-performing chatbots, users responded with lower-quality answers than others in the human-only setting. It seems that users would not simply ignore poor suggestions and compose responses as they could without seeing the suggestions. Besides, presetting the message box has improved reply efficiency without hurting response quality. We did not find that showing more suggestions helps or hurts response quality or efficiency consistently. Our study reveals how and when AI chatbot suggestions can help people answer questions in hybrid conversational systems.Keywords
Funding Information
- Wisconsin Alumni Research Foundation
This publication has 42 references indexed in Scilit:
- Smart Reply2016
- Towards Conversational Recommender Systems2016
- Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System2016
- A Cost-Benefit Study of Text Entry Suggestion Interaction2016
- How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation2016
- Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning2016
- Deep Reinforcement Learning for Dialogue Generation2016
- The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems2015
- The Hidden Information State model: A practical framework for POMDP-based spoken dialogue managementComputer Speech & Language, 2010
- Partially observable Markov decision processes for spoken dialog systemsComputer Speech & Language, 2007