Sharmila Upadhyaya

Shared tasks are powerful tools for advancing research through community-based standardised evaluation. As such, they play a key role in promoting findable, accessible, interoperable, and reusable (FAIR), as well as transparent and reproducible research practices. This paper presents an updated overview of twelve shared tasks developed and hosted under the German National Research Data Infrastructure for Data Science and Artificial Intelligence (NFDI4DS) consortium, covering a diverse set of challenges in scholarly document processing. Hosted at leading venues, the tasks foster methodological innovations and contribute open-access datasets, models, and tools for the broader research community, which are integrated into the consortium's research data infrastructure.

As research in Artificial Intelligence and Data Science continues to grow in volume and complexity, it becomes increasingly difficult to ensure transparency, reproducibility, and discoverability. To address these challenges, as research artifacts should be understandable and usable by machines, the NFDI4DataScience consortium is developing and providing Research Knowledge Graphs (RKGs). Building upon earlier works, this paper presents recent progress in creating semantically rich RKGs using standardized ontologies, shared vocabularies, and automated Information Extraction techniques. Key achievements include the development of the NFDI4DS ontology, metadata standards, tools, and services designed to support the FAIR principles, as well as community-led projects and various implementations of RKGs. Together, these efforts aim to capture and connect the complex relationships between datasets, models, software, and scientific publications.

Sharmila Upadhyaya, Wolfgang Otto, Frank Krüger, Stefan Dietze. Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025). 2025.

This paper describes our participation in the Shared Task on Software Mentions Disambiguation (SOMD), with a focus on improving relation extraction in scholarly texts through generative Large Language Models (LLMs) using single-choice question-answering. The methodology prioritises the use of in-context learning capabilities of LLMs to extract software-related entities and their descriptive attributes, such as distributive information. Our approach uses Retrieval-Augmented Generation (RAG) techniques and LLMs for Named Entity Recognition (NER) and Attributive NER to identify relationships between extracted software entities, providing a structured solution for analysing software citations in academic literature. The paper provides a detailed description of our approach, demonstrating how using LLMs in a single-choice QA paradigm can greatly enhance IE methodologies. Our participation in the SOMD shared task highlights the importance of precise software citation practices and showcases our system’s ability to overcome the challenges of disambiguating and extracting relationships between software mentions. This sets the groundwork for future research and development in this field.

BACKGROUND Recent advances in large language models (LLMs) have shown remarkable performance on various downstream tasks in zero- and few-shot scenarios, shedding light on named entity recognition (NER) in low-resource domains. Traditional Chinese medicine (TCM) against COVID-19 has been a new research topic and led to niche research literature. NER techniques are crucial for extracting and utilizing the rich knowledge in such literature. OBJECTIVE To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature. METHODS We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NER performance of LLMs can be assessed. We then performed NER tasks for the 6 entity types using ChatGPT (GPT-3.5 and GPT-4) and 4 state-of-the-art BERT-based question-answering (QA) models (RoBERTa, MiniLM, PubMedBERT and SciBERT) without prior training on the specific task. A domain fine-tuned model (GSAP-NER) was also applied for a comprehensive comparison on one of the entity types. In the task setup, we considered two different matching methods, namely exact match and fuzzy match, to have better coverage of potential application scenarios. The evaluation metrics we used are Precision, Recall, and F-1 in both exact match and fuzzy match. RESULTS The overall performance of LLMs varied significantly in exact match and fuzzy match. In the fuzzy match, ChatGPT surpassed BERT-based QA models in 5 out of 6 tasks, while in exact match, BERT-based QA models outperformed ChatGPT in 5 out of 6 tasks but with a smaller F-1 difference. GPT-4 showed a significant advantage over other models in fuzzy match, especially on the entity type of TCM formula and the Chinese patent drug (TFD) and ingredient (IG), achieving a higher F-1 score of 0.814 and 0.689, respectively. Although GPT-4 outperformed BERT-based models on entity type of herb (HB, 0.324), target (TG, 0.319), and research method (RM, 0.443), none of the F-1 scores exceeded 0.5. GSAP-NER, outperformed GPT-4 in terms of F-1 by a slight margin (0.45 vs 0.433) on RM. ChatGPT achieved considerably higher recalls than precisions, particularly in the fuzzy match. CONCLUSIONS The NER performance of LLMs is highly dependent on the entity type, and their performance varies across application scenarios. ChatGPT could be a good choice for scenarios where high recall is favored, e.g., for domain novices obtaining an extensive overview of the field. However, for knowledge acquisition in rigorous scenarios, neither ChatGPT nor BERT-based QA models are off-the-shelf tools for professional practitioners. Besides, these BERT-based models are open-source, and thus more accessible for scientific inquiry and hence worth further exploration.