Agentic Audio Moderator vs Human Moderator in Think-Aloud Usability Testing: Results from a Randomized Controlled Trial
Open Access
- 13 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-19
Abstract
Agentic AI holds promise for usability testing, yet its role as an audio moderator in think-aloud protocols is not well understood. This study explores: (1) how to design and develop an agentic audio moderator for think-aloud usability testing, and (2) how participants moderated by an agentic moderator differ from those moderated by a human regarding task performance, verbalization behaviors, user experience, and social perceptions of the moderator. Using a design-based research approach, we interviewed nine UX experts, iteratively developed an AI moderator, and evaluated it in a randomized controlled trial (N = 60) with a note-taking application. Results suggest that significant differences were not observed between AI and human moderators in task performance or verbalization behaviors, though AI moderators received lower social perception ratings. This work contributes the first design-oriented evaluation of AI moderators in usability testing, offering implications for developing more acceptable and effective agentic audio moderators.Keywords
Funding Information
- The Hong Kong Polytechnic University (A0056446)
This publication has 50 references indexed in Scilit:
- Anthropomorphic Enhancement Model Based on Large Language Models2025
- Digitalisation and the Work Environment: Insights from Evaluating the Contextual Think-Aloud Method2024
- Demystifying ChatGPT: An In-depth Survey of OpenAI’s Robust Large Language ModelsArchives of Computational Methods in Engineering, 2024
- Godspeed Questionnaire Series: Translations and Usage2023
- Comparison of Moderated and Unmoderated Remote Usability Sessions for Web-Based Simulation Software: A Randomized Controlled Trial2022
- Designing for Complementarity: Teacher and Student Needs for Orchestration Support in AI-Enhanced Classrooms2019
- An investigation of the efficacy of collaborative virtual reality systems for moderated remote usability testingApplied Ergonomics, 2017
- Protocol Analysis2017
- Self‐disclosure in computer‐mediated communication: The role of self‐awareness and visual anonymityEuropean Journal of Social Psychology, 2001
- Toward a Design Science of Education1992