Silvia Regina Vergilio

Search-Based Software Engineering (SBSE) tools offer automatic support by applying search-based techniques to solve different optimization problems of various software engineering areas. However, configuring SBSE tools often requires expertise in optimization techniques, posing a significant barrier for non-specialist users. This task involves selecting suitable algorithms, tuning parameters, choosing appropriate objective functions and search operators, and making decisions that affect optimization quality and performance. To address this challenge, we propose a multiple-LLM approach that integrates Large Language Models (LLMs), specifically ChatGPT-4, into SBSEworkflows to provide context-aware and natural language suggestions for parameter configuration. The approach follows a Divide and Conquer strategy, in which specialized LLMs are trained for distinct parameter configuration subproblems. As a proof of concept, we implemented OPLA-Wizard that works with OPLA-Tool v2.0, for Product Line Architecture (PLA) design. This assistant supports users in configuring algorithm settings, search operators, and objective functions through a user-friendly interface. Preliminary results show that the multiple-LLM approach improves accuracy, explanation clarity, and output format adherence compared to a single-LLM approach. This work contributes to the democratization of SBSE by reducing cognitive load, preventing misconfigurations, and making advanced optimization techniques more accessible to a broader audience.

Search-based design of Product Line Architecture (PLA) focuses on enhancing the design and functionality of software product lines through variability management, reuse, and optimization. A particular challenge in this area is the selection of objective functions, which significantly influence the success of the search process. Moreover, many objectives make the analysis and choice of a solution to be used harder. The literature has assigned this task to the PLA designer, i.e., the Decision-Maker (DM), who does not always know all the functions and their impact on the optimization outcomes. On the other hand, recent research shows that Large Language Models (LLMs), particularly the Generative Pre-trained Transformer series (GPT), have obtained promising results to help in various Software Engineering (SE) tasks. Considering this fact, this work explores the integration of such LLMs, notably ChatGPT, into the search-based PLA design. By leveraging LLMs’ capacity to understand/generate human-like text, we investigate their potential to assist DMs and propose an approach for suggesting objective functions, thereby simplifying and improving decision-making in PLA design optimization. Through empirical tests and qualitative feedback from domain experts, this research highlights the application of LLMs in search-based SE. The results demonstrate that integrating ChatGPT into PLA design can significantly enhance decision-making efficiency and solution quality, with a 40% reduction in time required for selecting objective functions and a 25% improvement in solution quality from the DM’s point of view. This study maps out the challenges and opportunities that lie ahead in fully harnessing their potential for PLA search-based design.

Test Case Prioritization (TCP) has proved to be beneficial when time constraints exist. During TCP, most relevant test cases are executed first in order to maximize failure detection and produce rapid feedback. There are several TCP approaches in the literature, but most of them do not consider particularities of Highly-Configurable Systems (HCSs), namely the notion of features. Existing work is usually based on artifacts (e.g., feature models) that can be outdated or are not available. Some code-oriented approaches usually employ dynamic analysis or require a failure-history, which can be costly. Furthermore, they present limitations when new variants and test cases are added during the system evolution. To address such limitations, we propose and evaluate three feature-oriented TCP strategies: (i) FeatChgHist: prioritizes test cases associated to features that often changed over commits; (ii) FeatChgCommit: prioritizes test cases associated to features that changed more in the current commit; and (iii) FeatCov: prioritizes test cases that cover more features simultaneously. Features are the building blocks of HCSs that are used for communication and design purposes. Then the basis to introduce these strategies is the fact that test cases associated to a great number of features, or to features that changed often over commits, are more fault-prone. In the evaluation, conducted with a real open-source HCS, FeatChgHist and FeatChgCommit show better performance regarding failure detection than FeatCov, and also than the changed-file-oriented strategies, used as baseline. Moreover, combining test selection and prioritization allows a reduced test set without decreasing significantly early failure detection.

As software evolves, new artifacts are created, modified, or removed. One of these main artifacts generated in the development of object-oriented software is the class. Classes have a very dynamic life cycle that can result in additional costs to the project. One way to mitigate this is to detect, in the early stages of the development, classes that are prone to change. Some approaches in the literature adopt Machine Learning (ML) algorithms to predict the change-proneness of a class. However, most of these approaches do not consider the temporal dependency between training instances, i.e., they consider that the instances are independent. To overcome such a limitation, this study presents an approach for predicting change-proneness based on the class change history. The approach adopts the sliding window method and is evaluated to obtain six kinds of models, which are derived by using, as predictors, different sets of metrics: structural, evolutionary, and smell-based. The evaluation uses five systems, four ML algorithms, and also explores some resample techniques to deal with imbalanced data. Regardless of the kind of model analyzed and the algorithm used, our approach overcomes the traditional one in 378 (~80) cases, out of 420, considering all systems, kinds of models, indicators, and algorithms. Moreover, the results show that our approach presents the best performance when the set of evolutionary metrics is used as predictors. There is no improvement when smell-based metrics are added. The Random Forest algorithm with the resampling technique ADA reaches the best performance among the ML algorithms evaluated.

Identifying which parts of code are prone to change during software evolution allows developers to prioritize and allocate resources efficiently. Having as focus a smaller scope makes easier change management and allows monitoring the type of modification and its impact. However, existing change-proneness prediction approaches are focused mainly on system classes. But the problem is that classes contain many characteristics of different software attributes and some software behaviors are more granular and better captured at the method-level. Motivated by these facts, in this paper, we empirically assess the performance of four machine learning algorithms for change-prone method prediction in seven open-source software projects. We derived and compared models obtained with three sets of independent variables (features): a set composed of structural metrics, a second set composed of evolution-based metrics, and a third that includes a combination of both kinds of metrics. The results show that, Random Forest presents the best general performance, independently of the used indicator and set of features. The model composed by both sets of metrics outperforms the other two. Two features based on the frequency of changes that happened in the evolution history of the method are point out as the most important for our problem.

In this paper, we present PerfoRT, a tool to ease software performance regression measurement of Java systems. Its main characteristics include: minimal configuration to ease automation and hide complexity to the end user; a broad scope of performance metrics including system, process, JVM, and tracing; and presentation of the results from a developer's perspective. We show some of its features in a usage example using Apache Commons BCEL project.

Continuous Integration (CI) environments is a practice adopted by most organizations that allows frequent integration of software changes, making software evolution more rapid and cost-effective. Such environments require dynamic Test Case Prioritization (TCP) approaches that adapt better to the test budgets and frequent addition/removal of test cases. In this sense, Ranking-to-Learn approaches have been proposed and are more suitable for CI constraints. By observing past prioritizations and guided by reward functions, they learn the best prioritization for a given commit. In order to contribute for improvements and direct future research, this work evaluates how far the solutions produced by these approaches are from optimal solutions produced by a deterministic approach (ground truth). To this end, we consider two learning-based approaches i) RETECS, which is based on Reinforcement Learning; and ii) COLEMAN, an approach based on Multi-Armed Bandit. The evaluation was conducted with twelve systems, three test budgets, two reward functions, and six measures concerning fault detection effectiveness, early fault detection, test time reduction in the CI cycles, prioritization time, and accuracy. Our findings have some implications for the approaches application and reward function choice. The approaches are applicable in real scenarios and produce solutions very close to the optimal ones, respectively, in 92% and 75% of the cases. Both approaches have some limitations to learn with few historical test data (a small number of CI Cycles) and deal with a large test case set, in which many failures are distributed over many test cases.