Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable Systems

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Test Case Prioritization (TCP) has proved to be beneficial when time constraints exist. During TCP, most relevant test cases are executed first in order to maximize failure detection and produce rapid feedback. There are several TCP approaches in the literature, but most of them do not consider particularities of Highly-Configurable Systems (HCSs), namely the notion of features. Existing work is usually based on artifacts (e.g., feature models) that can be outdated or are not available. Some code-oriented approaches usually employ dynamic analysis or require a failure-history, which can be costly. Furthermore, they present limitations when new variants and test cases are added during the system evolution. To address such limitations, we propose and evaluate three feature-oriented TCP strategies: (i) FeatChgHist: prioritizes test cases associated to features that often changed over commits; (ii) FeatChgCommit: prioritizes test cases associated to features that changed more in the current commit; and (iii) FeatCov: prioritizes test cases that cover more features simultaneously. Features are the building blocks of HCSs that are used for communication and design purposes. Then the basis to introduce these strategies is the fact that test cases associated to a great number of features, or to features that changed often over commits, are more fault-prone. In the evaluation, conducted with a real open-source HCS, FeatChgHist and FeatChgCommit show better performance regarding failure detection than FeatCov, and also than the changed-file-oriented strategies, used as baseline. Moreover, combining test selection and prioritization allows a reduced test set without decreasing significantly early failure detection.
Funding Information
  • Fundação Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (8887.464736/2019-00)
  • Conselho Nacional de Desenvolvimento Científico e Tecnológico (310034/2022-1)
  • A Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (PDR-10)

This publication has 47 references indexed in Scilit: