Open Access
- 22 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (ISSTA) , 206-228
Abstract
Despite advances in automated testing, manual testing remains prevalent due to the high maintenance demands associated with test script fragility—scripts often break with minor changes in application structure. Recent developments in Large Language Models (LLMs) offer a potential alternative by powering Autonomous Web Agents (AWAs) that can autonomously interact with applications. These agents may serve as Autonomous Test Agents (ATAs), potentially reducing the need for maintenance-heavy automated scripts by utilising natural language instructions similar to those used by human testers. This paper investigates the feasibility of adapting AWAs for natural language test case execution and how to evaluate them. We contribute with (1) a benchmark of three offline web applications, and a suite of 113 manual test cases, split between passing and failing cases, to evaluate and compare ATAs performance, (2) SeeAct-ATA and pinATA, two open-source ATA implementations capable of executing test steps, verifying assertions and giving verdicts, and (3) comparative experiments using our benchmark that quantifies our ATAs effectiveness. Finally we also proceed to a qualitative evaluation to identify the limitations of PinATA, our best performing implementation. Our findings reveal that our simple implementation, SeeAct-ATA, does not perform well compared to our more advanced PinATA implementation when executing test cases (50% performance improvement). However, while PinATA obtains around 60% of correct verdict and up to a promising 94% specificity, we identify several limitations that need to be addressed to develop more resilient and reliable ATAs, paving the way for robust, low maintenance test automation.Keywords
This publication has 44 references indexed in Scilit:
- Exploring Browser Automation: A Comparative Study of Selenium, Cypress, Puppeteer, and Playwright2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Erratum: Leveraging Flexible Tree Matching to repair broken locators in web automation scriptsInformation and Software Technology, 2022
- An Improving Approach for DOM-Based Web Test Suite Repair2021
- Generating and selecting resilient and maintainable locators for Web automated testingSoftware Testing, Verification and Reliability, 2021
- Three Open Problems in the Context of E2E Web Testing and a Vision: NEONATE2019
- Card-sorting2016
- Visual vs. DOM-Based Web Locators: An Empirical Study2014
- Automating Web navigation with the WebVCRComputer Networks, 2000
- Experimentation in Software Engineering2000