Will AI also replace inspectors? Investigating the potential of generative AIs in usability inspection
- 4 November 2025
- conference paper
- Published by Sociedade Brasileira de Computação
- p. 1-11
Abstract
Usability inspection is a well-established technique for identifying interaction issues in software interfaces, thereby contributing to improved product quality. However, it is a costly process that requires time and specialized knowledge from inspectors. With advances in Artificial Intelligence (AI), new opportunities have emerged to support this task, particularly through generative models capable of interpreting interfaces and performing inspections more efficiently. This study examines the performance of generative AIs in identifying usability problems, comparing them to those of experienced human inspectors. A software prototype was evaluated by four specialists and two AI models (GPT-4o and Gemini 2.5 Flash), using metrics such as precision, recall, and F1-score. While inspectors achieved the highest levels of precision and overall coverage, the AIs demonstrated high individual performance and discovered many novel defects, but with a higher rate of false positives and redundant reports. The combination of AIs and human inspectors produced the best results, revealing their complementarity. These findings suggest that AI, in its current stage, cannot replace human inspectors but can serve as a valuable augmentation tool to improve efficiency and expand defect coverage. The results provide evidence based on quantitative analysis to inform the discussion on the role of AI in usability inspections, pointing to viable paths for its complementary use in software quality assessment contexts.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation2025
- A Comparative Heuristic Evaluation of Kadi4Mat Through Human Evaluators and GPT-42024
- ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design2024
- Conversation-based hybrid UI for the repertory grid technique: A lab experiment into automation of qualitative surveysInternational Journal of Human-Computer Studies, 2024
- On the evaluation of usability design guidelines for improving network monitoring tools interfacesJournal of Systems and Software, 2022
- Usability inspection: Novice crowd inspectors versus expertJournal of Systems and Software, 2022
- A comparison of usability evaluation methods for evaluating e-commerce websitesBehaviour & Information Technology, 2012
- Experimentation in Software Engineering2012
- Heuristic evaluation: Comparing ways of finding and reporting usability problemsInteracting with Computers, 2007
- Understanding Inspection Methods: Lessons from an Assessment of Heuristic Evaluation2001