Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough
Preprint
Abstract
Conducting usability testing like cognitive walkthrough (CW) can be costly. Recent developments in large language models (LLMs), with visual reasoning and UI navigation capabilities, present opportunities to automate CW. We explored whether LLMs (GPT-4 and Gemini-2.5-pro) can simulate human behavior in CW by comparing their walkthroughs with human participants. While LLMs could navigate interfaces and provide reasonable rationales, their behavior differed from humans. LLM-prompted CW achieved higher task completion rates than humans and followed more optimal navigation paths, while identifying fewer potential failure points. However, follow-up studies demonstrated that with additional prompting, LLMs can predict human-identified failure points, aligning their performance with human participants. Our work highlights that while LLMs may not replicate human behaviors exactly, they can be leveraged for scaling usability walkthroughs and providing UI insights, offering a valuable complement to traditional usability testing.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation2025
- Mapping Natural Language Instructions to Mobile UI Action Sequences2020
- Mapping natural language commands to web elements2018
- Interrater agreement and interrater reliability: Key concepts, approaches, and applicationsResearch in Social and Administrative Pharmacy, 2013
- State of the Art on the Cognitive Walkthrough Method, Its Variants and EvolutionsInternational Journal of Human–Computer Interaction, 2010