The reliability of acceptability judgments across languages
Open Access
- 5 January 2018
- journal article
- research article
- Published by Open Library of the Humanities in Glossa: A Journal of General Linguistics
- Vol. 3 (1)
Abstract
The reliability of acceptability judgments made by individual linguists has often been called into question. Recent large-scale replication studies conducted in response to this criticism have shown that the majority of published English acceptability judgments are robust. We make two observations about these replication studies. First, we raise the concern that English acceptability judgments may be more reliable than judgments in other languages. Second, we argue that it is unnecessary to replicate judgments that illustrate uncontroversial descriptive facts; rather, candidates for replication can emerge during formal or informal peer review. We present two experiments motivated by these arguments. Published Hebrew and Japanese acceptability contrasts considered questionable by the authors of the present paper were rated for acceptability by a large sample of naive participants. Approximately half of the contrasts did not replicate. We suggest that the reliability of acceptability judgments, especially in languages other than English, can be improved using a simple open review system, and that formal experiments are only necessary in controversial cases.Keywords
This publication has 38 references indexed in Scilit:
- Quantitative methods in syntax/semantics research: A response to Sprouse and Almeida (2013)Language and Cognitive Processes, 2013
- Quantitative methods alone are not enough: Response to Gibson and FedorenkoTrends in Cognitive Sciences, 2010
- Relax, lean back, and be a linguistZeitschrift für Sprachwissenschaft, 2009
- Revisiting Satiation: Evidence for an Equalization Response StrategyLinguistic Inquiry, 2009
- Data and grammar: Means and individualsTheoretical Linguistics, 2007
- Linguistics and empirical evidenceTrends in Cognitive Sciences, 2003
- Construct states at the PF interfaceLinguistic Variation Yearbook, 2001
- Binding of Indeterminate Pronouns and Clause Structure in JapaneseLinguistic Inquiry, 2001
- Additional-wh effects and the adjunction site theoryJournal of East Asian Linguistics, 1994
- Dative questions: A study in the relation of acceptability to grammaticality of an english sentence typeCognition, 1973