K-ON: Stacking Knowledge on the Head Layer of Large Language Model
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (11) , 11745-11753
Abstract
Recent advancements in large language models (LLMs) have significantly improved various natural language processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP tasks. However, in knowledge graph (KG) scenarios, entities are the fundamental units and identifying an entity requires at least several tokens. This leads to a granularity mismatch between KGs and natural languages. To address this issue, we propose K-ON, which integrates KG knowledge into the LLM by employing multiple head layers for next k-step prediction. K-ON can not only generate entity-level results in one step, but also enables contrastive loss against entities, which is the most powerful tool in KG representation learning. Experimental results show that K-ON outperforms state-of-the-art methods that incorporate text and even the other modalities.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- KICGPTv2: Large Language Model With Knowledge in Context for Knowledge Graph CompletionIEEE Transactions on Knowledge and Data Engineering, 2026
- Exploring Large Language Models for Knowledge Graph Completion2025
- Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-Based RetrofittingProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Unifying Large Language Models and Knowledge Graphs: A RoadmapIEEE Transactions on Knowledge and Data Engineering, 2024
- MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid2023
- Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding2023
- Fusing topology contexts and logical rules in language models for knowledge graph completionInformation Fusion, 2022
- Multimodal Data Enhanced Representation Learning for Knowledge Graphs2019
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learningNeural Networks, 2018
- Convolutional 2D Knowledge Graph EmbeddingsProceedings of the AAAI Conference on Artificial Intelligence, 2018