Generative Text Modeling through Short Run Inference
Preprint
Abstract
Latent variable models for text, when trained successfully, accurately model the data distribution and capture global semantic and syntactic features of sentences. The prominent approach to train such models is variational autoencoders (VAE). It is nevertheless challenging to train and often results in a trivial local optimum where the latent variable is ignored and its posterior collapses into the prior, an issue known as posterior collapse. Various techniques have been proposed to mitigate this issue. Most of them focus on improving the inference model to yield latent codes of higher quality. The present work proposes a short run dynamics for inference. It is initialized from the prior distribution of the latent variable and then runs a small number (e.g., 20) of Langevin dynamics steps guided by its posterior distribution. The major advantage of our method is that it does not require a separate inference model or assume simple geometry of the posterior distribution, thus rendering an automatic, natural and flexible inference engine. We show that the models trained with short run dynamics more accurately model the data, compared to strong language model and VAE baselines, and exhibit no sign of posterior collapse. Analyses of the latent space show that interpolation in the latent space is able to generate coherent sentences with smooth transition and demonstrate improved classification over strong baselines with latent features from unsupervised pretraining. These results together expose a well-structured latent space of our generative model.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Effective Estimation of Deep Generative Language Models2020
- A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text2019
- Generating Sentences from a Continuous Space2016
- A large annotated corpus for learning natural language inference2015
- Annealed importance samplingStatistics and Computing, 2001
- Long Short-Term MemoryNeural Computation, 1997