Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)
Open Access
- 19 April 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-15
Abstract
AI is promising in assisting UX evaluators with analyzing usability tests, but its judgments are typically presented as non-interactive visualizations. Evaluators may have questions about test recordings, but have no way of asking them. Interactive conversational assistants provide a Q&A dynamic that may improve analysis efficiency and evaluator autonomy. To understand the full range of analysis-related questions, we conducted a Wizard-of-Oz design probe study with 20 participants who interacted with simulated AI assistants via text or voice. We found that participants asked for five categories of information: user actions, user mental model, help from the AI assistant, product and task information, and user demographics. Those who used the text assistant asked more questions, but the question lengths were similar. The text assistant was perceived as significantly more efficient, but both were rated equally in satisfaction and trust. We also provide design considerations for future conversational AI assistants for UX evaluation.Keywords
All Related Versions
This publication has 55 references indexed in Scilit:
- Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversationsComputers in Human Behavior, 2015
- Using psycho-physiological measures to assess task difficulty in software development2014
- Is once enough?2014
- Analysis in practical usability evaluation2012
- What do usability evaluators do in practice?2006
- The Evaluator Effect: A Chilling Fact About Usability Evaluation MethodsInternational Journal of Human–Computer Interaction, 2003
- The evaluator effect in usability tests1998
- Heuristic Walkthroughs: Finding the Problems Without the NoiseInternational Journal of Human–Computer Interaction, 1997
- Distributed collaborative writing1994
- Usability Heuristics1993