Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space
Preprint
Abstract
Generating high-resolution, photo-realistic images has been a long-standing goal in machine learning. Recently, Nguyen et al. (2016) showed one interesting way to synthesize novel images by performing gradient ascent in the latent space of a generator network to maximize the activations of one or multiple neurons in a separate classifier network. In this paper we extend this method by introducing an additional prior on the latent code, improving both sample quality and sample diversity, leading to a state-of-the-art generative model that produces high quality images at higher resolutions (227x227) than previous generative models, and does so for all 1000 ImageNet categories. In addition, we provide a unified probabilistic interpretation of related activation maximization methods and call the general class of models "Plug and Play Generative Networks". PPGNs are composed of 1) a generator network G that is capable of drawing a wide range of image types and 2) a replaceable "condition" network C that tells the generator what to draw. We demonstrate the generation of images conditioned on a class (when C is an ImageNet or MIT Places classification network) and also conditioned on a caption (when C is an image captioning network). Our method also improves the state of the art of Multifaceted Feature Visualization, which generates the set of synthetic inputs that activate a neuron in order to better understand how deep neural networks operate. Finally, we show that our model performs reasonably well at the task of image inpainting. While image models are used in this paper, the approach is modality-agnostic and can be applied to many types of data.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Cooperative Training of Descriptor and Generator NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Places: An Image Database for Deep Scene UnderstandingJournal of Vision, 2017
- Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network2017
- Alternating Back-Propagation for Generator NetworkProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Inception-v4, Inception-ResNet and the Impact of Residual Connections on LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Context Encoders: Feature Learning by Inpainting2016
- Show and tell: A neural image caption generator2015
- Optimal Scaling of Discrete Approximations to Langevin DiffusionsJournal of the Royal Statistical Society Series B: Statistical Methodology, 1998
- Exponential Convergence of Langevin Distributions and Their Discrete ApproximationsBernoulli, 1996