Counter-intuitive: Large Language Models Can Better Understand Knowledge Graphs Than We Thought
Preprint
Abstract
Although the method of enhancing large language models' (LLMs') reasoning ability and reducing their hallucinations through the use of knowledge graphs (KGs) has received widespread attention, the exploration of how to enable LLMs to integrate the structured knowledge in KGs on-the-fly remains inadequate. Researchers often co-train KG embeddings and LLM parameters to equip LLMs with the ability of comprehending KG knowledge. However, this resource-hungry training paradigm significantly increases the model learning cost and is also unsuitable for non-open-source, black-box LLMs. In this paper, we employ complex question answering (CQA) as a task to assess the LLM's ability of comprehending KG knowledge. We conducted a comprehensive comparison of KG knowledge injection methods (from triples to natural language text), aiming to explore the optimal prompting method for supplying KG knowledge to LLMs, thereby enhancing their comprehension of KG. Contrary to our initial expectations, our analysis revealed that LLMs effectively handle messy, noisy, and linearized KG knowledge, outperforming methods that employ well-designed natural language (NL) textual prompts. This counter-intuitive finding provides substantial insights for future research on LLMs' comprehension of structured knowledge.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- 🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language ModelsComputational Linguistics, 2025
- The Power of Noise: Redefining Retrieval for RAG Systems2024
- The Role and Limitations of Large Language Models Such as ChatGPT in Clinical Settings and Medical JournalismRadiology, 2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding2022
- An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels2022
- KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base2022
- CokeBERT: Contextual knowledge selection and embedding towards enhanced pre-trained language modelsAI Open, 2021
- CoLAKE: Contextualized Language and Knowledge Embedding2020
- 7th Open Challenge on Question Answering over Linked Data (QALD-7)2017