Knowledge Graph as Pre-Training Corpus for Structural Reasoning via Multi-Hop Linearization
Open Access
- 27 December 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 7273-7283
Abstract
Large language models have demonstrated exceptional performance across various natural language processing tasks. However, their reliance on unstructured text corpora for pre-training limits their effectiveness in tasks requiring structured reasoning such as multi-hop question-answering. Knowledge Graphs provide a rich, structured source of relational data, offering an opportunity to enhance the reasoning capabilities of Large language models. In this paper, we propose a novel framework, Knowledge Graph as Pre-training Corpus (KGPC), which transforms knowledge graphs into text using a multi-hop linearization process. Unlike existing approaches that linearize singular triples, our method captures the interconnected nature of knowledge graphs by linking multiple triples across multiple hops, preserving their relational structure during the pre-training phase. This structured knowledge injection improves language models’ abilities to perform complex reasoning tasks.We evaluate our approach on multi-hop reasoning benchmarks, demonstrating significant performance gains over existing models, particularly in question-answering tasks. Our results highlight the potential of multi-hop linearization in enhancing the structural reasoning capacity of language models, reducing error propagation, and improving the integration of structured knowledge into language models.Keywords
Funding Information
- Korea Agency for Infrastructure Technology Advancement (KAIA) grant
- Ministry of Land, Infrastructure and Transport (RS-2023-00244769)
- Korea Ministry of Land, Infrastructure and Transport (MOLIT) as the Innovative Talent Education Program for Smart City
This publication has 19 references indexed in Scilit:
- DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language UnderstandingProceedings of the AAAI Conference on Artificial Intelligence, 2022
- SKILL: Structured Knowledge Infusion for Large Language Models2022
- KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language RepresentationTransactions of the Association for Computational Linguistics, 2021
- Entities as Experts: Sparse Memory Access with Entity Supervision2020
- Table-to-Text: Describing Table Region With Natural LanguageProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Table-to-Text Generation by Structure-Aware Seq2seq LearningProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Variational Reasoning for Question Answering With Knowledge GraphProceedings of the AAAI Conference on Artificial Intelligence, 2018
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- DrugBank: a comprehensive resource for in silico drug discovery and explorationNucleic Acids Research, 2006