An Empirical Analysis of UI-Based Flaky Tests
Authors:
- 1 May 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1585-1597
Abstract
Flaky tests have gained attention from the research community in recent years and with good reason. These tests lead to wasted time and resources, and they reduce the reliability of the test suites and build systems they affect. However, most of the existing work on flaky tests focus exclusively on traditional unit tests. This work ignores UI tests that have larger input spaces and more diverse running conditions than traditional unit tests. In addition, UI tests tend to be more complex and resource heavy, making them unsuited for detection techniques involving rerunning test suites multiple times.In this paper, we perform a study on flaky UI tests. We analyze 235 flaky UI test samples found in 62 projects from both web and Android environments. We identify the common underlying root causes of flakiness in the UI tests, the strategies used to manifest the flaky behavior, and the fixing strategies used to remedy flaky UI tests. The findings made in this work can provide a foundation for the development of detection and prevention techniques for flakiness arising in UI tests.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Toward understanding compiler bugs in GCC and LLVM2016
- Making System User Interactive Tests Repeatable: When and What Should We Control?2015
- Empirically revisiting the test independence assumption2014
- An Empirical Study of Bugs in Machine Learning Systems2012
- An empirical study on configuration errors in commercial and open source systems2011
- How do fixes become bugs?2011
- Learning from mistakes2008
- A comparison of software defects in database management systems and operating systems2003
- Defect type and its impact on the growth curve (software development)2002
- An empirical study of operating systems errorsACM SIGOPS Operating Systems Review, 2001