AI of Oz: Enhancing Wizard of Oz Studies in HCI with AI Assistance for Human Moderation
Open Access
- 13 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Abstract
The Wizard of Oz (WoZ) method is a common and popular approach for simulating interactive systems in Human-Computer Interaction (HCI). Running such studies is demanding for researchers because the human wizard must manage human–agent interactions in real time while keeping participants safe and the interaction natural. Many WoZ systems struggle to reproduce complex agent behaviours without minimal delays or heavy workload for the moderator. We introduce AI of Oz, a framework that uses large language models to support researchers by monitoring ongoing interactions, detecting sensitive moments, and suggesting contextually appropriate responses. In a study with 20 HCI-related researchers, the system improved participants’ ability to manage interactions and maintain control compared to a version without AI support. We outline implications for WoZ research and note current limitations and future directions.Keywords
Funding Information
- MBIE Catalyst (3726177)
This publication has 64 references indexed in Scilit:
- Unveiling security, privacy, and ethical concerns of ChatGPTJournal of Information and Intelligence, 2023
- The risks of using ChatGPT to obtain common safety-related information and adviceSafety Science, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- My Chatbot Companion - a Study of Human-Chatbot RelationshipsInternational Journal of Human-Computer Studies, 2021
- AI in mental healthCurrent Opinion in Psychology, 2020
- CoZ: A crowd-powered system for social roboticsSoftwareX, 2020
- In‐the‐loop or on‐the‐loop? Interactional arrangements to support team coordination with a planning agentConcurrency and Computation: Practice and Experience, 2017
- Evaluation and Validation of a Conversational Agent Embodied in a BookstoreLecture Notes in Computer Science, 2003
- Wizard of Oz studies — why and howKnowledge-Based Systems, 1993
- Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research1988