Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
Preprint
Abstract
With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMsNatural Language Engineering, 2019
- Visualizing and Understanding the Effectiveness of BERT2019
- Do latent tree learning models identify meaningful structure in sentences?Transactions of the Association for Computational Linguistics, 2018
- The estimation of stochastic context-free grammars using the Inside-Outside algorithmComputer Speech & Language, 1990