Syntactic Structure Distillation Pretraining For Bidirectional Encoders
Preprint
Abstract
Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests of syntactic competence. Given this success, it remains an open question whether scalable learners like BERT can become fully proficient in the syntax of natural language by virtue of data scale alone, or whether they still benefit from more explicit syntactic biases. To answer this question, we introduce a knowledge distillation strategy for injecting syntactic biases into BERT pretraining, by distilling the syntactically informative predictions of a hierarchical---albeit harder to scale---syntactic language model. Since BERT models masked words in bidirectional context, we propose to distill the approximate marginal distribution over words in context from the syntactic LM. Our approach reduces relative error by 2-21% on a diverse set of structured prediction tasks, although we obtain mixed results on the GLUE benchmark. Our findings demonstrate the benefits of syntactic biases, even in representation learners that exploit large amounts of data, and contribute to a better understanding of where syntactic biases are most helpful in benchmarks of natural language understanding.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference2019
- AllenNLP: A Deep Semantic Natural Language Processing Platform2018
- LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better2018
- Assessing the Ability of LSTMs to Learn Syntax-Sensitive DependenciesTransactions of the Association for Computational Linguistics, 2016
- Globally Normalized Transition-Based Neural Networks2016
- Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear ModelsComputational Linguistics, 2007
- Tri-training: exploiting unlabeled data using three classifiersIEEE Transactions on Knowledge and Data Engineering, 2005
- The Proposition Bank: An Annotated Corpus of Semantic RolesComputational Linguistics, 2005
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Head-driven Phrase Structure Grammar. By Carl Pollard and Ivan A. Sag Chicago & London: The University of Chicago Press, 1994. Pp. xi, 440.Language, 1996