Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests
Open Access
- 20 March 2021
- book chapter
- Published by Springer Nature
- Vol. 12651, 270-287
Abstract
Software developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministically pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.Keywords
This publication has 26 references indexed in Scilit:
- Modeling and ranking flaky tests at Apple2020
- Root causing flaky tests in a large-scale industrial setting2019
- From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis2018
- Practical Test Dependency Detection2018
- What Causes My Test Alarm? Automatic Cause Analysis for Test Alarms in System and Integration Testing2017
- Efficient dependency detection for safe Java test acceleration2015
- Reliable testing: detecting state-polluting tests to prevent test dependency2015
- Empirically Detecting False Test Alarms Using Association Rules2015
- The Art of Testing Less without Sacrificing Quality2015
- Improving oracle quality by detecting brittle assertions and unused inputs in tests2014