Kai Petersen

Automated unit test generation using Large Language Models (LLMs) has shown promise for widely used programming languages such as Java and Python. It may further benefit legacy systems lacking a test base, and the investigation of less widespread legacy languages remains relevant as many companies still operate systems built on them. We develop a modular workflow combining static code analysis, multi-stage prompt engineering with business logic extraction, automated error analysis, and iterative test-fix cycles for legacy systems, evaluated by pass rates, coverage, and repair effectiveness. We evaluate our approach on four productive IBM Perl repositories (186 functions) and three CPAN distributions (139 functions) using two open-source LLMs: Llama-3.3-70B-Instruct and Mistral-Small-3.1-24B-Instruct-2503. A key methodological contribution is a defensive wrapper that records a crash in the code under test as a failed assertion rather than terminating the run. A git-diff mode enables selective test generation for changed functions, supporting incremental maintenance workflows. The larger model achieves initial pass rates of 66–82% (industrial) and 74–81% (CPAN), improving to 74–86% and 76–89% after repair, with statement coverage of 34–92% and 65–100%. The smaller model performs substantially worse on both sets. Repair reduces structural errors on both ( from 0–44% to 0–6% and 0–7% to 0–5%) and improves pass rates for both models in most repositories. We show the transferability of LLM-based test generation to low-resource legacy languages, establishing Perl as a viable target for automated testing pipelines.

Context: The state-of-the-art and practice on software quality is growing constantly and presents several challenges. The ever-growing body of knowledge on the topic, obfuscates the situation further, the lack of explicit structure makes it difficult to identify which properties exist today and how they can be evaluated, two critical aspects to increase software quality. Objective: A step to disambiguate software properties descriptions is made via a Property Model Ontology (PMO) and steps towards the evaluation of the structure are carried out together with experts. The objectives of this paper are: 1) present in detail the PMO, 2) describe the research process used to develop and evaluate the PMO, and, 3) exemplify the usage of the PMO through real instantiations obtained from practitioners and researchers. Method: Expert interviews and qualitative research methods are used to evaluate the PMO and develop a proof of concept. Results: The PMO consists of concepts describing extra-functional properties (EFPs) and their evaluation methods, i.e., how to measure the properties. The PMO is instantiated in a modelling environment through a metamodel and an online web-content management system. Conclusions: Consensus on the definition and structure of EFPs is achieved and a common understanding on how they can be reused in practice.

In the dynamic field of Software Engineering (SE), where practice is constantly evolving and adapting to new technologies, conducting research is a daunting quest. This poses a challenge for researchers: how to stay relevant and effective in their studies? Empirical Software Engineering (ESE) has emerged as a contending force aiming to critically evaluate and provide knowledge that informs practice in adopting new technologies. Empirical research requires a rigorous process of collecting and analyzing data to obtain evidence-based findings. Challenges to this process are numerous, and many researchers, novice and experienced, found difficulties due to many complexities involved in designing their research. The core of this chapter is to teach foundational skills in research design, essential for educating software engineers and researchers in ESE. It focuses on developing a well-structured research design, which includes defining a clear area of investigation, formulating relevant research questions, and choosing appropriate methodologies. While the primary focus is on research design, this chapter also covers aspects of research scoping and selecting research methods. This approach prepares students to handle the complexities of the ever-changing technological landscape in SE, making it a critical component of their educational curriculum.

Industrial relevance is essential for an applied research area like software engineering. However, it is unclear how to achieve industrial relevance and how we communicate and assess it. We propose a reasoning framework to support the design, reporting, and assessment of research for industrial relevance.

Background: Business intelligence provides decision support to decision-makers in public administrations. So far, researchers have studied the use of business intelligence systems concerning, for example, efficiency gains of the organization. However, based on our literature study, researchers have yet to focus on the impact of business intelligence systems on the efficiency of analytical processes in the public administration context. Objective: This study's objective is to investigate the impact of introducing a business intelligence system in public administration. Method: A case study was conducted to investigate the impact. We studied four departments at Flensburg Municipality in Germany. We used interviews and analyzed documentation to describe the analytical processes and to estimate efficiency gains.Results: The BI system led to efficiency gains across departments. The gains varied from 1.77% to 47%. Losses were observed regarding media changes.Conclusions: The impact on efficiency varies with the attributes of the analytical process (e.g., repetitions in the process). Hence, further investigations are needed to build theories regarding the relationship between efficiency and the use of BI systems in public administrations.

Replication studies help solidify and extend knowledge by evaluating previous studies’ findings. Software engineering literature showed that too few replications are conducted focusing on software artifacts without the involvement of humans. This study aims to replicate an artifact-based study on software testing to address the gap related to replications. In this investigation, we focus on (i) providing a step-by-step guide of the replication, reflecting on challenges when replicating artifact-based testing research and (ii) evaluating the replicated study concerning the validity and robustness of the findings. We replicate a test case prioritization technique proposed by Kwon et al. We replicated the original study using six software programs, four from the original study and two additional software programs. We automated the steps of the original study using a Jupyter notebook to support future replications. Various general factors facilitating replications are identified, such as (1) the importance of documentation; (2) the need for assistance from the original authors; (3) issues in the maintenance of open-source repositories (e.g., concerning needed software dependencies, versioning); and (4) availability of scripts. We also noted observations specific to the study and its context, such as insights from using different mutation tools and strategies for mutant generation. We conclude that the study by Kwon et al. is partially replicable for small software programs and could be automated to facilitate software practitioners, given the availability of required information. However, it is hard to implement the technique for large software programs with the current guidelines. Based on lessons learned, we suggest that the authors of original studies need to publish their data and experimental setup to support the external replications.

Regression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large‐scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal–question–metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.