R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling
Preprint
Abstract
Human language understanding operates at multiple levels of granularity (e.g., words, phrases, and sentences) with increasing levels of abstraction that can be hierarchically combined. However, existing deep models with stacked layers do not explicitly model any sort of hierarchical process. This paper proposes a recursive Transformer model based on differentiable CKY style binary trees to emulate the composition process. We extend the bidirectional language model pre-training objective to this architecture, attempting to predict each word given its left and right abstraction nodes. To scale up our approach, we also introduce an efficient pruned tree induction algorithm to enable encoding in just a linear number of composition steps. Experimental results on language modeling and unsupervised parsing show the effectiveness of our approach.Keywords
All Related Versions
- Version 1, 2021-07-02, ArXiv
- Version 2, 2022-03-03, ArXiv
- Published version: , 4897.
This publication has 18 references indexed in Scilit:
- Unsupervised Parsing with S-DIORA: Single Tree Encoding for Deep Inside-Outside Recursive Autoencoders2020
- Masked Language Model Scoring2020
- Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERT2020
- Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMsNatural Language Engineering, 2019
- Grammar Induction with Neural Language Models: An Unusual Replication2018
- Recurrent Neural Network Grammars2016
- The Forest Convolutional Network: Compositional Distributional Semantics with a Neural Chart and without Binarization2015
- Deep learning2014
- The Penn Chinese TreeBank: Phrase structure annotation of a large corpusNatural Language Engineering, 2005
- Three models for the description of languageIEEE Transactions on Information Theory, 1956