Leveraging LLMs for Automated Extraction and Structuring of Educational Concepts and Relationships
Open Access
- 19 September 2025
- journal article
- research article
- Published by MDPI AG in Machine Learning and Knowledge Extraction
- Vol. 7 (3) , 103
Abstract
Students must navigate large catalogs of courses and make appropriate enrollment decisions in many online learning environments. In this context, identifying key concepts and their relationships is essential for understanding course content and informing course recommendations. However, identifying and extracting concepts can be an extremely labor-intensive and time-consuming task when it has to be done manually. Traditional NLP-based methods to extract relevant concepts from courses heavily rely on resource-intensive preparation of detailed course materials, thereby failing to minimize labor. As recent advances in large language models (LLMs) offer a promising alternative for automating concept identification and relationship inference, we thoroughly investigate the potential of LLMs in automatically generating course concepts and their relations. Specifically, we systematically evaluate three LLM variants (GPT-3.5, GPT-4o-mini, and GPT-4o) across three distinct educational tasks, which are concept generation, concept extraction, and relation identification, using six systematically designed prompt configurations that range from minimal context (course title only) to rich context (course description, seed concepts, and subtitles). We systematically assess model performance through extensive automated experiments using standard metrics (Precision, Recall, F1, and Accuracy) and human evaluation by four domain experts, providing a comprehensive analysis of how prompt design and model choice influence the quality and reliability of the generated concepts and their interrelations. Our results show that GPT-3.5 achieves the highest scores on quantitative metrics, whereas GPT-4o and GPT-4o-mini often generate concepts that are more educationally meaningful despite lexical divergence from the ground truth. Nevertheless, LLM outputs still require expert revision, and performance is sensitive to prompt complexity. Overall, our experiments demonstrate the viability of LLMs as a tool for supporting educational content selection and delivery.Keywords
Funding Information
- JST SPRING (JPMJSP2136, JP20H00622, JP24K20903)
- Japan Society for the Promotion of Science (JPMJSP2136, JP20H00622, JP24K20903)
This publication has 19 references indexed in Scilit:
- Core Concept Identification in Educational Resources via Knowledge Graphs and Large Language ModelsSN Computer Science, 2024
- A survey on large language models for recommendationWorld Wide Web, 2024
- ChatGPT for Education Research: Exploring the Potential of Large Language Models for Qualitative Codebook Development2024
- Large Language Models for Recommendation: Progresses and Future Directions2023
- Detection of GPT-4 Generated Text in Higher Education: Combining Academic Judgement and Software to Identify Generative AI Tool MisuseJournal of Academic Ethics, 2023
- Weakly supervised setting for learning concept prerequisite relations using multi-head attention variational graph auto-encodersKnowledge-Based Systems, 2022
- Knowledge Graph-Based Core Concept Identification in Learning Resources2018
- Resources Sequencing Using Automatic Prerequisite--Outcome AnnotationACM Transactions on Intelligent Systems and Technology, 2015
- Identifying core concepts in educational resources2012
- Term-weighting approaches in automatic text retrievalInformation Processing & Management, 1988