Monique Snoeck

The breakthrough of Large Language Models (LLMs) has changed how different kinds of complex tasks are approached, including the ones that require a higher level of abstraction and critical thinking together with advanced domain-specific knowledge, like Conceptual Modeling. Several experiments on testing the modeling capabilities of LLMs have already been conducted, but the literature still lacks a structured analysis of how different LLMs and prompting techniques impact the extraction of conceptual models, as UML class diagrams, from textual specifications. In this paper, we present a comprehensive comparison of open-source and closed-source LLMs used in conjunction with the most effective and accessible prompting techniques, on a newly crafted high-quality dataset of case specifications, implementing an automated evaluation on generated UML class diagrams. Finally, we assess how factors like model size or case complexity impact the quality of the generated models and what LLM and what prompting technique to choose for which task. The dataset and the experimental source code are made available through GitHub( https://github.com/IlKaiser/text2uml ).

Developers spend a large portion of their time on software comprehension, making it essential to understand how different artifacts – models and code – support this activity. While prior research has primarily explored the use of models as complementary aids for understanding code, limited evidence exists regarding their standalone effectiveness in conveying software’s structural and behavioral aspects. This gap is particularly relevant in contexts such as Model-Driven Engineering (MDE) and Low-Code/No-Code (LCNC) approaches, where models often serve as the primary or sole representation of software. The paper reports on two complementary controlled experiments that investigate how models and code contribute to understanding software structure and behavior. The first experiment compares comprehension when only reverse-engineered UML models or source code were available (code-centric settings), mimicking scenarios in agile or AI-assisted development where models are outdated or absent. The second experiment focuses on model-driven settings, examining whether providing generated code alongside platform-independent models improves comprehension and whether differences in code generators influence performance. The results demonstrate that models and code achieve comparable overall correctness, and thus models alone may be sufficient in model-centric scenarios where access to code is limited or unavailable. The experiments also reveal a consistent structure-behavior comprehension gap, show differences in comprehension across tasks regardless of artifact type, and indicate that adding code to models can both clarify and confuse comprehension depending on context.

Context: Modern software development increasingly relies on conceptual models as blueprints for system design, particularly in low-code/no-code and Model-Driven Engineering approaches. Ensuring the semantic quality of these models is non-trivial, and testing-based approaches can be used for this.Objectives: One particularly promising testing-based approach is model mutation testing, which applies code-based mutation testing principles to conceptual models. In this approach, models with minor syntactic differences from the original model are created and tested against the test suite. However, model mutation testing requires skills that students often lack. Gamification has been proposed as a solution to this problem, as it structures complexity through game mechanics. Therefore, we investigated whether gamified model mutation testing improves students' testing skills.Method: We developed ModelDefenders, a competitive web game for entity-relationship model testing based on model mutation testing, and conducted a pretest-treatment-posttest experiment with students (n=34) enrolled in varying university degrees.Results: The results show that students created higher quality test cases post-treatment. The tool received positive user ratings independent of personal characteristics.Conclusion: We conclude that the gamified approach used in our research is associated with higher model testing performance, providing empirical evidence for gamification effectiveness in software engineering education.

The data-driven study and optimization of learning processes has become possible through collecting the digital traces left by learners as they interact with digital study materials and learning tools — a field known as Learning Analytics (LA). The increased availability of such data has fueled the growth of the LA field, enabling the development of new frameworks and models, and supporting the transfer of findings across domains, and collaboration on joint educational initiatives. However, given the sensitive and personal nature of intermediary learning results and outcomes, such data availability also raises significant ethical and privacy concerns. To support and foster the open and transparent development and evaluation of LA solutions, we present a detailed clickstream dataset collected at KU Leuven across two first-year bachelor courses over three academic years. The public dataset is accompanied by transparent documentation of the de-identification process, and we report on privacy and utility validation results.