A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement
Preprint
Abstract
We present a novel dataset for physical and abstract plausibility of events in English. Based on naturally occurring sentences extracted from Wikipedia, we infiltrate degrees of abstractness, and automatically generate perturbed pseudo-implausible events. We annotate a filtered and balanced subset for plausibility using crowd-sourcing, and perform extensive cleansing to ensure annotation quality. In-depth quantitative analyses indicate that annotators favor plausibility over implausibility and disagree more on implausible events. Furthermore, our plausibility dataset is the first to capture abstractness in events to the same extent as concreteness, and we find that event abstractness has an impact on plausibility ratings: more concrete event participants trigger a perception of implausibility.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Do Neural Language Models Overcome Reporting Bias?2020
- How Relevant Are Selectional Preferences for Transformer-based Language Models?2020
- Is Wikipedia succeeding in reducing gender bias? Assessing changes in gender bias in Wikipedia using word embeddings2020
- Inherent Disagreements in Human Textual InferencesTransactions of the Association for Computational Linguistics, 2019
- Learning Supervised Topic Models for Classification and Regression from CrowdsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Truth Is a Lie: Crowd Truth and the Seven Myths of Human AnnotationAI Magazine, 2015
- Reporting bias and knowledge acquisition2013
- A Flexible, Corpus-Driven Model of Regular and Inverse Selectional PreferencesComputational Linguistics, 2010
- Beyond perceptual symbols: A call for representational pluralismCognition, 2009
- A preferential, pattern-seeking, Semantics for natural language inferenceArtificial Intelligence, 1975