Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
Preprint
Abstract
The Winograd Schema (WS) has been proposed as a test for measuring commonsense capabilities of models. Recently, pre-trained language model-based approaches have boosted performance on some WS benchmarks but the source of improvement is still not clear. This paper suggests that the apparent progress on WS may not necessarily reflect progress in commonsense reasoning. To support this claim, we first show that the current evaluation method of WS is sub-optimal and propose a modification that uses twin sentences for evaluation. We also propose two new baselines that indicate the existence of artifacts in WS benchmarks. We then develop a method for evaluating WS-like sentences in a zero-shot setting to account for the commonsense reasoning abilities acquired during the pretraining and observe that popular language models perform randomly in this setting when using our more strict evaluation. We conclude that the observed progress is mostly due to the use of supervision in training WS models, which is not likely to successfully support all the required commonsense reasoning skills and knowledge.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Measuring and Improving Consistency in Pretrained Language ModelsTransactions of the Association for Computational Linguistics, 2021
- An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language ModelsTransactions of the Association for Computational Linguistics, 2020
- oLMpics-On What Language Model Pre-training CapturesTransactions of the Association for Computational Linguistics, 2020
- BLiMP: The Benchmark of Linguistic Minimal Pairs for EnglishTransactions of the Association for Computational Linguistics, 2020
- What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language ModelsTransactions of the Association for Computational Linguistics, 2020
- PIQA: Reasoning about Physical Commonsense in Natural LanguageProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Evaluating Commonsense in Pre-Trained Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2020
- The Sensitivity of Language Models and Humans to Winograd Schema Perturbations2020
- Breaking NLI Systems with Sentences that Require Simple Lexical Inferences2018
- Acquiring Comparative Commonsense Knowledge from the WebProceedings of the AAAI Conference on Artificial Intelligence, 2014