TRAC: A Textual Benchmark for Reasoning about Actions and Change
Preprint
Abstract
Reasoning about actions and change (RAC) is essential to understand and interact with the ever-changing environment. Previous AI research has shown the importance of fundamental and indispensable knowledge of actions, i.e., preconditions and effects. However, traditional methods rely on logical formalization which hinders practical applications. With recent transformer-based language models (LMs), reasoning over text is desirable and seemingly feasible, leading to the question of whether LMs can effectively and efficiently learn to solve RAC problems. We propose four essential RAC tasks as a comprehensive textual benchmark and generate problems in a way that minimizes the influence of other linguistic requirements (e.g., grounding) to focus on RAC. The resulting benchmark, TRAC, encompassing problems of various complexities, facilitates a more granular evaluation of LMs, precisely targeting the structural generalization ability much needed for RAC. Experiments with three high-performing transformers indicates that additional efforts are needed to tackle challenges raised by TRAC.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Implicit Representations of Meaning in Neural Language Models2021
- PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World2021
- Generalization in Instruction Following Systems2021
- PRover: Proof Generation for Interpretable Reasoning over Rules2020
- Know What You Don’t Know: Unanswerable Questions for SQuAD2018
- Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension2018
- Combining Natural Logic and Shallow Reasoning for Question Answering2016
- Knowledge Representation and Reasoning2004
- Linguistics and Natural Logic1972
- Strips: A new approach to the application of theorem proving to problem solvingArtificial Intelligence, 1971