Florian Freund

Preparing training data for domain-specific medical Named Entity Recognition (NER) involves a trade-off between annotation quality, expert effort, and data privacy: manual annotation is costly, whereas cloud-based Large Language Models (LLMs) raise concerns about the control of sensitive clinical text. This article introduces NERFlow, a workflow-driven subsystem of the FIT4NER project whose contribution is an abstraction layer that makes rule-based, model-based, and LLM-based annotation interchangeable and comparable within one configurable workflow environment. Open-source LLMs are integrated as exchangeable annotation services, deployable locally or in cloud-agnostic infrastructures via Kubernetes, and embedded into the KM-EP knowledge management system. NERFlow was evaluated qualitatively, through a cognitive walkthrough, an IEEE 1028 technical review, and a user-centered survey with 18 participants, and quantitatively on the CRAFT corpus with seven open-source and hosted LLMs run through an identical pipeline. The results support its use as LLM-assisted pre-annotation with expert correction, with locally deployable open-source models as the more reliable basis for reproducible operation.

In this article, we present Stanford Named Entity Recognition and Classification (SNERC), an intelligent system designed to enhance knowledge management through named entity recognition (NER) and document classification (DC) in the field of Applied Gaming. In this domain, the effective application of NER and DC is essential for addressing information overload (IO), enabling software developers to efficiently search, filter, and retrieve large volumes of textual data from web sources. SNERC streamlines the management and deployment of machine learning (ML)-based NER models, supporting the accurate extraction of named entities (NEs) and the classification of heterogeneous textual documents. The system tackles key challenges in NER, such as the impact of language and domain specificity on model performance, domain adaptation, and the complexity of handling diverse NE types. We demonstrate SNERC’s capabilities through real-world use cases, highlighting improvements in DC and information retrieval (IR) within applied gaming scenarios. The system provides core functionalities for training, evaluating, and managing NER models using the Stanford CoreNLP framework. Additionally, SNERC integrates with a rule-based expert system (RBES) to enable the automatic categorization of documents into predefined taxonomies within a knowledge management system. We present results from comprehensive qualitative and quantitative evaluations—measured through precision, recall, and F-score—to assess the system’s effectiveness and identify areas for further optimization, supporting seamless integration into real-world operational environments. Received: 15 July 2024 | Revised: 9 April 2025 | Accepted: 29 April 2025 Conflicts of Interest The authors declare that they have no conflicts of interest to this work. Data Availability Statement Data sharing is not applicable to this article as no new data were created or analyzed in this study. Author Contribution Statement Philippe Tamla: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization. Florian Freund: Methodology, Software, Validation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization. Matthias Hemmje: Conceptualization, Validation, Supervision, Project administration.

This article focuses on assisting medical professionals in analyzing domain-specific texts and selecting and comparing Named Entity Recognition (NER) frameworks. It details the development and evaluation of a system that utilizes a generic approach alongside the structured Nunamaker methodology. This system empowers medical professionals to train, evaluate, and compare NER models across diverse frameworks, such as Stanford CoreNLP, spaCy, and Hugging Face Transformers, independent of their specific implementations. Additionally, it introduces a concept for modeling a general training and evaluation process. Finally, experiments using various ontologies from the CRAFT corpus are conducted to assess the effectiveness of the current prototype.

This paper presents a cloud-based system that builds upon the FIT4NER framework to support medical experts in training machine learning models for named entity recognition (NER) using Microsoft Azure. The system is designed to simplify complex cloud configurations while providing an intuitive interface for managing and converting large-scale training and evaluation datasets across formats such as PDF, DOCX, TXT, BioC, spaCyJSON, and CoNLL-2003. It also enables the configuration of transformer-based spaCy pipelines and orchestrates Azure cloud services for scalable and efficient NER model training. Following the structured Nunamaker research methodology, the paper introduces the research context, surveys the state of the art, and highlights key challenges faced by medical professionals in cloud-based NER. It then details the modeling, implementation, and integration of the system. Evaluation results—both qualitative and quantitative—demonstrate enhanced usability, scalability, and accessibility for non-technical users in medical domains. The paper concludes with insights gained and outlines directions for future work.

This paper presents a survey based on Kasunic's survey research methodology to identify the criteria used by Machine Learning (ML) experts to evaluate Named Entity Recognition (NER) tools and frameworks. Comparison and selection of NER tools and frameworks is a critical step in leveraging NER for Information Retrieval to support the development of Clinical Practice Guidelines. In addition, this study examines the main challenges faced by ML experts when choosing suitable NER tools and frameworks. Using Nunamaker's methodology, the article begins with an introduction to the topic, contextualizes the research, reviews the state-of-the-art in science and technology, and identifies challenges for an expert survey on NER tools and frameworks. This is followed by a description of the survey's design and implementation. The paper concludes with an evaluation of the survey results and the insights gained, ending with a summary and conclusions.

The increasing availability of vast quantities of data from various sources significantly impacts the insurance industry, although this industry has always been data driven. It accelerates manual processes and enables new products or business models. On the other hand, it also burdens insurance analysts and other users that need to cope with this development parallel to other global changes. A novel information system (IS) for artificial intelligence (AI)-supported big data analysis, introduced within this paper, shall help to overcome user overload and to empower human data analysts in the insurance industry. The IS research’s focus lies neither in novel algorithms nor datasets but in concepts that combine AI and big data analysis for synergies, such as usability enhancements. For this purpose, this paper systematically designs and implements an AI2VIS4BigData reference model to help information systems conform to automatically detect anomalies and increase its users’ confidence and efficiency. Practical relevance is assured by an interview with an insurance analyst to verify the demand for the developed system and derive all requirements from two insurance industry user stories. A core contribution is the introduction of the IS. Another significant contribution is an extension of the AI2VIS4BigData service-based architecture and user interface (UI) concept on AI and machine learning (ML)-based user empowerment and data transformation. The implemented prototype was applied to synthetic data to enable the evaluation of the system. The quantitative and qualitative evaluations confirm the system’s usability and applicability to the insurance domain yet reveal the need for improvements toward bigger quantities of data and further evaluations with a more extensive user group.

Climate change means coping directly or indirectly with extreme weather conditions for everybody. Therefore, analyzing meteorological data to create precise models is gaining more importance and might become inevitable. Meteorologists have extensive domain knowledge about meteorological data yet lack practical data analysis skills. This paper presents a method to bridge this gap by empowering the data knowledge carriers to analyze the data. The proposed system utilizes symbolic AI, a knowledge base created by experts, and a recommendation expert system to offer suiting data analysis methods or data pre-processing to meteorologists. This paper systematically analyzes the target user group of meteorologists and practical use cases to arrive at a conceptual and technical system design implemented in the CAMeRI prototype. The concepts in this paper are aligned with the AI2VIS4BigData Reference Model and comprise a novel first-order logic knowledge base that represents analysis methods and related pre-processings. The prototype implementation was qualitatively and quantitatively evaluated. This evaluation included recommendation validation for real-world data, a cognitive walkthrough, and measuring computation timings of the different system components.