BFBenedikt Fein

Benedikt Fein

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Benedikt Fein has a total of 11 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Reasoning About Bugs in Learners’ Scratch Programs Using Large Language Models

    by Benedikt Fein, Patric Feldmeier, Gordon Fraser, Florian Obermüller


  • Open Access

    Challenges of deploying code embeddings: an industrial case study on method name generation

    by Benedikt Fein, Maximilian Jungwirth, Gordon Fraser, Florian Kandlinger

    The recent hype around machine learning has fully captured software engineering research. Correspondingly, a variety of different ways to represent code as input to deep learning models have been proposed. These code embedding models are usually evaluated in terms of common metrics such as accuracy or bleu scores, on benchmark tasks such as predicting method names from their body. Although this evaluation approach is well established in research, it leaves open challenges for the deployment of these models in practice: First, comparing accuracy on standardised benchmark results conveniently avoids some of the challenges of actually running different prototype model implementations, which, however is necessary to apply the models in practice. Second, the models are usually trained and evaluated on abundantly available open-source training data, which may be very different from closed-source industrial code. Third, the deployment of machine learning models in an industrial environment does not only entail technical but also organisational challenges. Finally, while competitive accuracy or bleu scores may be indicative of relative model performance, they may not reflect to what extent the models are suitable for being used by developers. In this paper we describe our experience of evaluating and deploying state-of-research code embedding models in an industrial environment, and present lessons learned from our struggles with each of these questions.

  • Open Access

    Challenges of Deploying Code Embeddings in Industry

    by Benedikt Fein, Maximilian Jungwirth, Gordon Fraser, Florian Kandlinger

    The recent hype around machine learning has fully captured software engineering research. Correspondingly, a variety of different ways to represent code as input to deep learning models have been proposed. These code embedding models are usually evaluated in terms of common metrics such as accuracy or bleu scores, on benchmark tasks such as predicting method names from their body. Although this evaluation approach is well established in research, it leaves open challenges for the deployment of these models in practice: First, comparing accuracy on standardised benchmark results conveniently avoids some of the challenges of actually running different prototype model implementations, which, however is necessary to apply the models in practice. Second, the models are usually trained and evaluated on abundantly available open-source training data, which may be very different from closed-source industrial code. Third, the deployment of machine learning models in an industrial environment does not only entail technical but also organisational challenges. Finally, while competitive accuracy or bleu scores may be indicative of relative model performance, they may not reflect to what extent the models are suitable for being used by developers. In this paper we describe our experience of deploying and evaluating state-of-research code embedding models in an industrial environment, and present lessons learned from our struggles with each of these questions.

  • Open Access

    Multiclass Classification and Prioritisation of Static Analysis Warnings Using Developer-Labelled Industrial Data

    by Benedikt Fein, Vibhash Kumar Singh, Gordon Fraser

    Automatic static code analysis tools are used to identify code quality issues like vulnerabilities or performance problems. In practice, the high number of irrelevant warnings created by such tools is problematic, but can be addressed by pre-filtering and ranking these warnings before they are shown to the developer. Since ground truth labelled data is rarely available, existing research tends to heuristically construct training data from unlabelled open-source data by assigning clearly separated binary categories like fixed and irrelevant to the warnings. However, this labelling approach cannot capture distinctions such as between relevant, but not yet fixed and not fixed, because irrelevant warnings. The Teamscale software developed by CQSE provides a unique opportunity to investigate this concern, since its developers have adopted the practice of meticulously labelling all static analysis warnings as either accepted, tolerated, or false-positive. Using this dataset, we adapt previously proposed models for a new multiclass classification task to evaluate both their ability to classify warnings in this new setting and their ability to prioritise important warnings. Our experiments show that especially the mentioned subtle difference between categories of unresolved warnings is more challenging for the models compared to a binary prediction. Nevertheless, training the models for the multiclass task rather than the binary one results in a statistically significant improvement in the prioritisation of the warnings.

  • Open Access

    On the Applicability of Language Models to Block-Based Programs

    by Elisabeth Griebl, Benedikt Fein, Florian Obermüller, Gordon Fraser, René Just

    Block-based programming languages like Scratch are increasingly popular for programming education and end-user programming. Recent program analyses build on the insight that source code can be modelled using techniques from natural language processing. Many of the regularities of source code that support this approach are due to the syntactic overhead imposed by textual programming languages. This syntactic overhead, however, is precisely what block-based languages remove in order to simplify programming. Consequently, it is unclear how well this modelling approach performs on block-based programming languages. In this paper, we investigate the applicability of language models for the popular block-based programming language Scratch. We model Scratch programs using n-gram models, the most essential type of language model, and transformers, a popular deep learning model. Evaluation on the example tasks of code completion and bug finding confirm that blocks inhibit predictability, but the use of language models is nevertheless feasible. Our findings serve as foundation for improving tooling and analyses for block-based languages.