The Defeat of the Winograd Schema Challenge
Authors:
Preprint
Abstract
The Winograd Schema Challenge - a set of twin sentences involving pronoun reference disambiguation that seem to require the use of commonsense knowledge - was proposed by Hector Levesque in 2011. By 2019, a number of AI systems, based on large pre-trained transformer-based language models and fine-tuned on these kinds of problems, achieved better than 90% accuracy. In this paper, we review the history of the Winograd Schema Challenge and discuss the lasting contributions of the flurry of research that has taken place on the WSC in the last decade. We discuss the significance of various datasets developed for WSC, and the research community's deeper understanding of the role of surrogate tasks in assessing the intelligence of an AI system.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Word meaning in minds and machines.Psychological Review, 2023
- Human-level play in the game ofDiplomacyby combining language models with strategic reasoningScience, 2022
- Technical perspective: The importance of WINOGRANDECommunications of the ACM, 2021
- Using Answer Set Programming for Commonsense Reasoning in the Winograd Schema ChallengeTheory and Practice of Logic Programming, 2019
- Commonsense reasoning about containers using radically incomplete informationArtificial Intelligence, 2017
- Commonsense reasoning and commonsense knowledge in artificial intelligenceCommunications of the ACM, 2015
- On our best behaviourArtificial Intelligence, 2014
- Coherence and Coreference RevisitedJournal of Semantics, 2007
- Discourse coherence and pronoun resolutionLanguage and Cognitive Processes, 2004
- Artificial intelligence meets natural stupidityACM SIGART Bulletin, 1976