EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
Preprint
Abstract
Quantitative reasoning is a higher-order reasoning skill that any intelligent natural language understanding system can reasonably be expected to handle. We present EQUATE (Evaluating Quantitative Understanding Aptitude in Textual Entailment), a new framework for quantitative reasoning in textual entailment. We benchmark the performance of 9 published NLI models on EQUATE, and find that on average, state-of-the-art methods do not achieve an absolute improvement over a majority-class baseline, suggesting that they do not implicitly learn to reason with quantities. We establish a new baseline Q-REAS that manipulates quantities symbolically. In comparison to the best performing NLI model, it achieves success on numerical reasoning tests (+24.2%), but has limited verbal reasoning capabilities (-8.1%). We hope our evaluation framework will support the development of models of quantitative reasoning in language understanding.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- SciTaiL: A Textual Entailment Dataset from Science Question AnsweringProceedings of the AAAI Conference on Artificial Intelligence, 2018
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- Annotation Artifacts in Natural Language Inference Data2018
- Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems2017
- Parsing Algebraic Word Problems into EquationsTransactions of the Association for Computational Linguistics, 2015
- Reasoning about Quantities in Natural LanguageTransactions of the Association for Computational Linguistics, 2015
- Learning to Automatically Solve Algebra Word Problems2014
- The third PASCAL recognizing textual entailment challenge2007
- The Number Sense: How the Mind Creates Mathematics.The American Mathematical Monthly, 1998