Tony Gorschek

Context: The integration of generative artificial intelligence (GenAI) into software engineering (SE) activities creates a growing need for organizations to understand and evaluate its impact. However, measuring GenAI in practice remains challenging mainly due to the difficulty of interpreting productivity and quality outcomes in a meaningful manner. Objectives: In this manuscript, we report on study that aims to understand how organizations can design, conduct, and interpret the impact of GenAI in SE settings and to capture this understanding in the form of a context-aware measurement handbook. Method: With two industry partners, we followed a Design Science Research approach involving iterative cycles of problem conceptualization, handbook design, and practitioner evaluation. Our handbook was developed through literature synthesis and prior industrial experiences, and refined through validation feedback sessions with 9 practitioners from two industrial contexts in the telecommunications and industry automation domains. Results: The resulting handbook structures GenAI measurement along five key SE activities, three measurement focus areas, corresponding qualitative and quantitative metrics, data collection strategies, analyses approaches, and confoundings factors. In addition, the industrial evaluations revealed practical challenges and lessons learned related to the operationalization of GenAI measurements. Conclusions: Our findings show that evaluating GenAI in SE contexts demands moving beyond isolated productivity metrics toward more holistic and socio-technical measurement approaches. The proposed handbook and practitioner-informed lessons learned provide a guidance for organizations seeking to evaluate GenAI adoption and understand its impact within their specific engineering settings.

Generative Artificial Intelligence (GenAI) is increasinglyintegrated into software products to enable new features and user capabilities, from early exploration to operational deployment. GenAI adoption as a component within a software system introduces quality risks because GenAI outputs are probabilistic, prompt-sensitive, and may drift after release. Organizations, therefore, need to decide what to evaluate, when to evaluate, and who owns quality evaluation activities across software design, development, and operations. ISO/IEC 25059 standard distinguishes between software product quality (e.g., usability) and quality-in-use (e.g., satisfaction) for AI-enabled software, yet it provides limited operational guidance for these evaluation activities. We therefore investigate how industrial software teams adopt and use GenAI models in the software systems they build and operate, and how they evaluate system qualities when deciding to adopt GenAI during development and after deployment. We do not benchmark the underlying GenAI model itself. In this study, we conducted 19 semi-structured interviews in two software development companies. We triangulated the interviews with archival data (15 internal documents and 184 internal wiki/web pages) to capture GenAI adoption steps, quality concerns, evaluation practices, and role responsibilities. Our findings describe a three-phase adoption process – Ideation, Development, and Operation – highlighting where quality evaluations occur, which criteria are used, and how evaluation responsibilities are distributed. Based on observed practices and using ISO/IEC 25059 as an organizing lens, we synthesize a process-oriented quality evaluation framework. This framework maps metrics to explicit gatekeeping, validation, and monitoring checkpoints, bridging abstract ISO quality characteristics with engineering workflows. We applied the framework in a GenAI-enabled software product (SE4AI) use case and reported how it supported structured evaluation activities. We also observed that quality evaluations span legal, security, development, QA, and operations, but ownership is fragmented across phases. We therefore propose a GenAI Quality Lead responsibility (often assignable to an existing senior role) to coordinate criteria, evidence, and traceability across quality evaluation activities. The results contribute to Software Engineering for AI (SE4AI) by clarifying how teams can measure qualities when building software that adopts and uses GenAI.

This study empirically examines the drivers of firms’ perceived needs for digital competence, focusing on the role of digital maturity in shaping demand for digital skills. The results reveal divergence dynamics: digitally mature firms, particularly those have AI/automation capabilities, R&D investments, and highly educated workforces, are significantly more likely to anticipate future digital skill needs, especially in decision-making roles. Firm size, innovation orientation, and international exposure also positively influence competence demand, though with varying intensity. Our findings highlight that digital frontier firms are strengthening their strategic investment in digital competence and digital laggard risk falling further behind.

Context: Generative AI (GenAI) is increasingly adopted in software development for tasks such as document generation, data analysis, and code generation. However, evaluating the quality of GenAI applications becomes challenging, as traditional quality measurements may not be fully applicable. Objective: In this study, we explore how practitioners evaluate the quality of GenAI applications and investigate quality evaluation techniques. Method: We conducted a multi-case study in three industrial projects from software development companies. We examined four GenAI application domains: document generation, data analysis and insight generation, customer service, and code generation. Data were collected through three workshops and 23 semi-structured interviews with industrial practitioners. Results: We identified fourteen GenAI use cases and 28 metrics currently used to evaluate the quality of GenAI applications’ outputs. We synthesized the identified metrics’ usage patterns and challenges based on the collected data. Conclusions: This study presents practical insights into using metrics to measure GenAI-based system qualities in real industrial settings. Our findings indicate that practitioners use custom-built and context-specific metrics; combining these with academic metrics can strengthen GenAI system quality evaluation.

Background: System safety in aeronautics is critical, as it directly affects aircraft reliability, efficiency, safety, and security. Given the complexity of modern aviation systems and the potential consequences of failures, a structured and proactive safety approach is essential. System-Theoretic Process Analysis (STPA) is a modern hazard analysis method designed to identify and mitigate risks. Unlike traditional methods that focus primarily on component failures, STPA accounts for both failures and unsafe interactions among system elements, including human operators, software, and organizational factors. Problem: Despite its effectiveness, STPA poses challenges in practical application. The process is time-consuming and requires extensive expertise in system safety, control theory, and system dynamics. Analysts must heavily rely on expert judgment to define losses, hazards, safety constraints, and unsafe control actions. Additionally, training in STPA is resource-intensive, making automation an appealing solution to streamline the process. Goal: To address these challenges, we developed two AI-driven pipelines to automate the initial steps of STPA, reducing reliance on expert knowledge and enhancing efficiency. Method: The first pipeline leverages a fine-tuned Llama3.1-8B model to extract losses, hazards, and constraints from ConOps documents. The second pipeline, BERT Error Detection for STPA (BEDS), improves accuracy by classifying, verifying, detecting errors, and suggesting potential corrections for the extracted elements. Results: The first pipeline was trained using 134 ConOps documents paired with corresponding STPA safety analysis elements. The dataset comprised 35 authentic documents from the CORDIS repository and 99 AI-generated examples. The model achieved a mean precision of 79.73%, recall of 81.09%, and an F1-score of 80.22%. For the second pipeline, 1,084 sentences were extracted from values identified during the first step of STPA. Three classifiers were developed: the sentence identifier achieved a mean accuracy of 95.20%, the incorrect sentence detector 88.61%, and the sentence error identifier 83.44%. While the pipelines were designed to work together, they can also be used independently. Conclusion: This study tackles the challenges of applying STPA in aeronautics by introducing two automated pipelines to streamline the initial process steps. The first pipeline, powered by a fine-tuned Llama3.1-8B model, extracts losses, hazards, and constraints from ConOps documents. The second pipeline, BEDS, verifies and corrects these elements with high accuracy. The results demonstrate strong precision and recall scores, highlighting the potential to reduce both the time and expertise required for STPA analysis in complex aviation systems.

Context: Software engineering (SE) artifacts and documents, such as requirements specifications, user stories, test cases, and concepts of operations (ConOps), are typically written in natural language, making their manipulation challenging. Natural Language Processing (NLP) is a viable solution for managing these tasks. Objective: To conduct a systematic literature review to explore the current use of NLP in SE artifacts and tasks, supplemented by a tertiary study focusing on the emerging role of Large Language Models (LLMs) in software engineering research. Method: We searched digital libraries for relevant papers and applied inclusion and exclusion criteria to filter the primary studies. We then analyzed NLP techniques applied to SE documents and examined their usage in this context. Our research methodology followed Kitchenham and Charters' guidelines. Additionally, we conducted a tertiary study to synthesize findings from existing systematic literature reviews and surveys specifically addressing LLMs in software engineering. Results: We selected 60 primary studies to identify the most common methods for NLP pipelines, feature extraction, language models, and machine learning algorithms used in SE. We also assessed the purposes of these methods, their benefits for SE, their difficulty, and their contribution to SE advancement. The tertiary study revealed a rapid proliferation of LLM-focused research, with comprehensive reviews documenting exponential growth in publications and widespread adoption across diverse SE tasks. Conclusion: Requirements are the most frequently addressed artifacts using NLP techniques, with preprocessing and part-of-speech (POS) tagging being widely used. There is a notable increase in the use of large language models for various SE tasks, such as requirements elicitation, source code generation, bug fixing, and software testing. The tertiary study confirms that LLMs represent a pivotal shift in the research landscape, warranting dedicated investigation to understand their transformative impact on NLP applications in software engineering.