From Human- to LLM-Centered Collaborative Ontology Engineering
- 31 January 2025
- journal article
- Published by SAGE Publications in Applied Ontology
Abstract
In the continuously evolving landscape of knowledge engineering, the symbiosis and teaming of humans and machines emerge as a pivotal new domain. This article explores the multifaceted realms of human and machine collaborative ontology engineering (OE). The goal of the presented work is to explore the potential of Large Language Models (LLMs) to speed up and automate the processes of collaborative OE, experimenting with different levels of LLM involvement. The proposed approach is based on a human-centered approach, that is, the HCOME approach to collaborative OE, and follows a process of exploring the declining involvement of humans and the parallel increase of LLM involvement, concluding at a level of automation where the OE is exclusively performed by LLMs. This experimentation is organized based on a series of human/LLM collaboration levels (a spectrum of OE), each one aligned to a specific OE methodology, that is, Level-0 HCOME (Human), Level-1 X-HCOME (Human and LLMs), Level-2 SimX-HCOME (LLMs and Human), and Level-3 Sim-HCOME (LLMs). The evaluation of these methodologies (one per level) is performed by measuring the similarity of the generated ontologies against “reference” ontologies (precision, recall, and F1-score of reference-to-LLM-generated ontological mappings). The results presented in this paper demonstrate that while LLMs significantly expedite the OE process, the accuracy and completeness of the resulting ontologies are notably enhanced by maintaining a high level of human involvement. This study is expected to contribute to a deeper understanding of evolving dynamics in LLM-based/enhanced OE, paving the way for future advancements toward more effective collaborative OE frameworks.Keywords
This publication has 13 references indexed in Scilit:
- Improving the use of LLMs in radiology through prompt engineering: from precision prompts to zero-shot learningRöFo - Fortschritte auf dem Gebiet der Röntgenstrahlen und der bildgebenden Verfahren, 2024
- Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES): a method for populating knowledge bases using zero-shot learningBioinformatics, 2024
- Generative Agents: Interactive Simulacra of Human Behavior2023
- LLMs4OL: Large Language Models for Ontology Learning2023
- When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task2023
- Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems2023
- Ontology engineering methodologies for the evolution of living and reused ontologies: status, trends, findings and recommendationsThe Knowledge Engineering Review, 2020
- Using the Ontology Maturing Process Model for Searching, Managing and Retrieving Resources with Semantic Technologies2008
- Lightweight Community-Driven Ontology Evolution2007
- Human-centered ontology engineering: The HCOME methodologyKnowledge and Information Systems, 2005