Unifying Structure Reasoning and Language Pre-Training for Complex Reasoning Tasks
- 19 October 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 32 (23299290) , 1586-1595
Abstract
Recent pre-trained language models (PLMs) equipped with foundation reasoning skills have shown remarkable performance on downstream complex tasks. However, the significant structure reasoning skill has been rarely studied, which involves modeling implicit structure information within the text and performing explicit logical reasoning over them to deduce the conclusion. This paper proposes a unified learning framework that combines explicit structure reasoning and language pre-training to endow PLMs with the structure reasoning skill. It first identifies several elementary structures within contexts to construct structured queries and performs step-by-step reasoning along the queries to identify the answer entity. The fusion of textual semantics and structure reasoning is achieved by using contextual representations learned by PLMs to initialize the representation space of structures, and performing stepwise reasoning on this semantic representation space. Experimental results on four datasets demonstrate that the proposed model achieves significant improvements in complex reasoning tasks involving diverse structures, and shows transferability to downstream tasks with limited training data and effectiveness for complex reasoning of KGs modality.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (6217020551)
- Science and Technology Commission of Shanghai Municipality (21QA1400600)
This publication has 33 references indexed in Scilit:
- From LSAT: The Progress and Challenges of Complex ReasoningIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- oLMpics-On What Language Model Pre-training CapturesTransactions of the Association for Computational Linguistics, 2020
- A Knowledge-Enhanced Pretraining Model for Commonsense Story GenerationTransactions of the Association for Computational Linguistics, 2020
- SpanBERT: Improving Pre-training by Representing and Predicting SpansTransactions of the Association for Computational Linguistics, 2020
- Strengthening spatial reasoning: elucidating the attentional and neural mechanisms associated with mental rotation skill developmentCognitive Research: Principles and Implications, 2020
- A Hybrid Deep Learning Approach for Spatial Trigger Extraction from Radiology Reports2020
- TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments2019
- Constructing Datasets for Multi-hop Reading Comprehension Across DocumentsTransactions of the Association for Computational Linguistics, 2018
- Neuroscience-Inspired Artificial IntelligenceNeuron, 2017
- How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversityJournal of Research in Science Teaching, 2009