Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
Preprint
Abstract
We investigate large-scale latent variable models (LVMs) for neural story generation -- an under-explored application for open-domain long text -- with objectives in two threads: generation effectiveness and controllability. LVMs, especially the variational autoencoder (VAE), have achieved both effective and controllable generation through exploiting flexible distributional latent representations. Recently, Transformers and its variants have achieved remarkable effectiveness without explicit latent representation learning, thus lack satisfying controllability in generation. In this paper, we advocate to revive latent variable modeling, essentially the power of representation learning, in the era of Transformers to enhance controllability without hurting state-of-the-art generation effectiveness. Specifically, we integrate latent representation vectors with a Transformer-based pre-trained architecture to build conditional variational autoencoder (CVAE). Model components such as encoder, decoder and the variational posterior are all built on top of pre-trained language models -- GPT2 specifically in this paper. Experiments demonstrate state-of-the-art conditional generation ability of our model, as well as its excellent representation learning capability and controllability.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta PosteriorProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space2020
- PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable2020
- Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Multi-space Variational Encoder-Decoders for Semi-supervised Labeled Sequence Transduction2017
- Generating Sentences from a Continuous Space2016
- Auto-association by multilayer perceptrons and singular value decompositionBiological Cybernetics, 1988