Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs
Authors:
- Ghazi FelhiGFGhazi Felhi
- Joseph Le RouxJRJoseph Le Roux
- Djamé SeddahDSDjamé Seddah
Preprint
Abstract
We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers. In the attention of Transformers, keys handle information selection while values specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values. We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syntax transfer capabilities when compared to supervised models and that comparable supervised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Interpretable machine learning: Fundamental principles and 10 grand challengesStatistics Surveys, 2022
- A Primer in BERTology: What We Know About How BERT WorksTransactions of the Association for Computational Linguistics, 2020
- Overestimation of Syntactic Representation in Neural Language Models2020
- A Systematic Assessment of Syntactic Generalization in Neural Language Models2020
- Do Neural Language Models Show Preferences for Syntactic Formalisms?2020
- Building a User-Generated Content North-African Arabizi Treebank: Tackling Hell2020
- Learning under Concept Drift: A ReviewIEEE Transactions on Knowledge and Data Engineering, 2018
- Processing of S-cone signals in the inner plexiform layer of the mammalian retinaVisual Neuroscience, 2013
- Effectiveness of Early Educational InterventionScience, 2011
- Revising the evolutionary computation abstraction2010